Faster learning via optimised adaboost

Deep Convolutional Neural Networks (CNNs) perform billions of operations for classifying a single input. To reduce these computations, this paper offers a solution that leverages a combination of runtime information and the algorithmic structure of CNNs. Specifically, in numerous modern CNNs, the outputs of compute-heavy convolution operations are fed to activation units that output zero if their input is negative. By exploiting this unique algorithmic property, we propose a predictive early activation technique, dubbed SnaPEA. This technique cuts the computation of convolution operations short if it determines that the output will be negative. SnaPEA can operate in two distinct modes, exact and predictive. In the exact mode, with no loss in classification accuracy, SnaPEA statically re-orders the weights based on their signs and periodically performs a single-bit sign check on the partial sum. Once the partial sum drops below zero, the rest of computations can simply be ignored, since the output value will be zero in any case. In the predictive mode, which trades the classification accuracy for larger savings, SnaPEA speculatively cuts the computation short even earlier than the exact mode. To control the accuracy, we develop a multi-variable optimization algorithm that thresholds the degree of speculation. As such, the proposed algorithm exposes a knob to gracefully navigate the trade-offs between the classification accuracy and computation reduction. Compared to a state-of-the-art CNN accelerator, SnaPEA in the exact mode, yields, on average, 28% speedup and 16% energy reduction in various modern CNNs without affecting their classification accuracy. With 3% loss in classification accuracy, on average, 67.8% of the convolutional layers can operate in the predictive mode. The average speedup and energy saving of these layers are 2.02x and 1.89x, respectively. The benefits grow to a maximum of 3.59x speedup and 3.14x energy reduction. Compared to static pruning approaches, which are complimentary to the dynamic approach of SnaPEA, our proposed technique offers up to 63% speedup and 49% energy reduction across the convolution layers with no loss in classification accuracy.

In this work we present key elements of DeepChip, a framework that bridges recent trends in machine learning with applicable forward propagation on resource-constrained devices. Main objective of this work is to reduce compute and memory requirements by removing redundancy from neural networks. DeepChip features a flexible quantizer to reduce the bit width of activations to 8-bit fixed-point and weights to an asymmetric ternary representation. In combination with novel algorithms and data compression we leverage reduced precision and sparsity for efficient forward propagation on a wide range of processor architectures. We validate our approach on a set of different convolutional neural networks and datasets: ConvNet on SVHN, ResNet-44 on CIFAR10 and AlexNet on ImageNet. Compared to single-precision floating point, memory requirements can be compressed by a factor of 43, 22 and 10 and computations accelerated by a factor of 5.2, 2.8 and 2.0 on a mobile processor without a loss in classification accuracy. DeepChip allows trading accuracy for efficiency, and for instance tolerating about 2% loss in classification accuracy further reduces memory requirements by a factor of 88, 29 and 13, and speeds up computations by a factor of 6.0, 4.3 and 5.0. Code related to this paper is available at: https://github.com/UniHD-CEG/ECML2018.

Deep learning (DL) has shown outstanding performances in many fields, including remote sensing (RS). DL is turning into an essential tool for the RS research community. Recently, many cloud platforms have been developed to provide access to large-scale computing capacity, consequently permitting the usage of DL architectures as a service. However, this opened the door to new challenges associated with the privacy and security of data. The RS data used to train the DL algorithms have several privacy requirements. Some of them need a high level of confidentiality, such as satellite images related to public security with high spatial resolutions. Moreover, satellite images are usually protected by copyright, and the owner may strictly refuse to share them. Therefore, privacy-preserving deep learning (PPDL) techniques are a possible solution to this problem. PPDL enables training DL on encrypted data without revealing the original plaintext. This study proposes a hybrid PPDL approach for object classification for very-high-resolution satellite images. The proposed encryption scheme combines Paillier homomorphic encryption (PHE) and somewhat homomorphic encryption (SHE). This combination aims to enhance the encryption of satellite images while ensuring a good runtime and high object classification accuracy. The method proposed to encrypt images is maintained through the public keys of PHE and SHE. Experiments were conducted on real-world high-resolution satellite images acquired using the SPOT6 and SPOT7 satellites. Four different CNN architectures were considered, namely ResNet50, InceptionV3, DenseNet169, and MobileNetV2. The results showed that the loss in classification accuracy after applying the proposed encryption algorithm ranges from 2% to 3.5%, with the best validation accuracy on the encrypted dataset reaching 92%.

Pattern recognition involves identifying a new instance as a member of a class based on the features extracted from problem space. Not all features contribute to increase in accuracy of classification. Wrapper based methodology selects an optimal subset of reduced features with no much loss in classification accuracy. Inspired by the bio evolutionary behaviour of flowers in optimal reproduction, a wrapper based optimal feature selection methodology is devised. In this paper, a Binary Flower Pollination Algorithm (BFPA) is proposed to address the combinatorial problem of feature selection. Support Vector machine (SVM) is used as a subset evaluator to calculate the fitness function to be optimized. Proposed algorithm is evaluated using 5 datasets from UCI repository. Switching probability is varied to achieve a fair tradeoff between exploration and exploitation. Feature subset chosen by the proposed method serves as input to 6 traditional classifiers to analyse the success of the algorithm. Experimental results are also compared with existing literature and observed that the proposed methodology identifies feature subset with reduced features and improved accuracy.

This paper introduces an automated method for estimating sex from the lower and upper limbs based on diaphyseal CSG properties. The proposed method was developed and evaluated using 389 femurs, 412 tibias, and 404 humeri of adult individuals from a modern Greek reference sample, the Athens Collection. The skeletal properties, which were extracted with the CSG-Toolkit, were analyzed with step-wise DFA (evaluated with LOOCV) and subsequently with RBF kernel SVM supervised learning. SVM cross-validation was based on a 20-fold stratified random sample splitting as well as a chronological split based on year of birth to further assess the effect of secular change in sex estimation capacity. Maximum cross-validated classification accuracy from step-wise DFA reached 94.8% for the femur, 94.7% for the tibia, and 97.3% for the humerus, whereas SVM cross-validated results were similar although slightly lower, mainly due to the more strict cross-validation scheme. Our results suggest that the proposed sex estimation method is reasonably robust to secular change, since there was limited loss in classification accuracy between different chronological groups, despite the presence of secular change in stature of the Greek population during the examined period. The proposed method has been implemented as a function for the GNU Octave environment, named estimate_sex, which comprises a self-intuitive graphical user interface for facilitating sex estimation and is freely available under a suitable license.

New instruments currently in design for Earth observational remote sensing from space offer promising advancements in the use of remote sensing for soil studies. Due to the enormous quantities of data generated by these high spectral resolution sensors, efficient use will only be feasible if new algorithms are developed to compress the data while maintaining information content. A spectral band selection algorithm is proposed to identify the important spectral bands for compression and classification of soil reflectance data. This algorithm was developed from the shape dominancy concept of Karhunen‐Loeve based optimal features. The high dimensional raw data were first transformed into much lower dimensionality by the derived spectral features. Canonical analysis was then used to transform the data under the maximal separability criterion into their final signal space where the data classification was performed. Soil data sets with and without stratification by soil order and climatic moisture zone were used to test the algorithm. The probabilities of correct classification of soil organic matter content using Landsat Multispectral Scanner (MSS) bands, Thematic Mapper (TM) bands, and the bands identified using the new band selection algorithm, were calculated and compared. The algorithm was successful in finding important spectral bands for soil organic matter content classification. The probabilities of correct classification obtained by using these bands were 0.850 and 0.883 for unstratified data sets, as compared to 0.600 to 0.640 for MSS bands and 0.640 to 0.653 for TM bands. Probabilities of correct classification for climate‐stratified data ranged from 0.910 to 0.980 using the calculated bands. The overall data compression ratio achieved using the algorithm was greater than 10 with no loss in classification accuracy.

We propose a hierarchical stimulus presentation paradigm for a P300-based Hangul (Korean script) input system. A P300-based input system is one of the most promising noninvasive brain-computer interface (BCI) applications based on its direct applicability in many computer programs. While the previous row/column stimulus presentation paradigm has been well-suited to the English input, it may not be optimal for a Hangul input because Hangul has a distinct hierarchical structure. To overcome the limitation of the previous paradigms, we developed a new P300-based Hangul input system by taking the unique hierarchical structure of Hangul into account for creating a hierarchical stimulus presentation paradigm. By using the hierarchical structure, we can effectively reduce the window size of the interface without loss of classification accuracy. A performance comparison shows that the hierarchical paradigm exhibits higher classification accuracy than the row/column paradigm even with a smaller window size. Thus the proposed hierarchical paradigm is more efficient to spell Hangul and will be more useful for BCI-based Hangul input for a text messenger, e-mail program, word processor, and so on.

In the context of support vector machines (SVM), high dimensional input vectors often reduce the computational efficiency and significantly slow down the classification process. In this paper, we propose a strategy to rank individual components according to their influence on the class assignments. This ranking is used to select an appropriate subset of the features. It replaces the original feature set without significant loss in classification accuracy. Often, the generalization ability of the classifier even increases due to the implicit regularization achieved by feature pruning.

Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. An important factor contributing to the long training times is the increasing dataset complexity required to reach state-of-the-art performance in real-world applications. To address this challenge, we explore the use of input mixing, where multiple inputs are combined into a single composite input with an associated composite label for training. The goal is for training on the mixed input to achieve a similar effect as training separately on each the constituent inputs that it represents. This results in a lower number of inputs (or mini-batches) to be processed in each epoch, proportionally reducing training time. We find that naive input mixing leads to a considerable drop in learning performance and model accuracy due to interference between the forward/backward propagation of the mixed inputs. We propose two strategies to address this challenge and realize training speedups from input mixing with minimal impact on accuracy. First, we reduce the impact of inter-input interference by exploiting the spatial separation between the features of the constituent inputs in the network's intermediate representations. We also adaptively vary the mixing ratio of constituent inputs based on their loss in previous epochs. Second, we propose heuristics to automatically identify the subset of the training dataset that is subject to mixing in each epoch. Across ResNets of varying depth, MobileNetV2 and two Vision Transformer networks, we obtain upto 1.6 × and 1.8 × speedups in training for the ImageNet and Cifar10 datasets, respectively, on an Nvidia RTX 2080Ti GPU, with negligible loss in classification accuracy.

Traffic classification using statistical characteristics (or fingerprints) of IP flows such as packet size and packet inter-arrival time has showed its preliminary success in sense of accuracy and simplicity. However, the need of large memory to store the fingerprints makes it impractical to deploy such method on backbone networks where a high-speed memory is needed to catch up with the high packet rate, and a large high-speed memory is always expensive. In this paper we apply the Distributional Clustering (DC) algorithm proposed by the pattern recognition community to compress the protocol fingerprints. We also presented a new algorithm named Distributional Quantification (DQ) that has lower overhead than DC. We evaluated the accuracy of classification using compact protocol fingerprints under various compression ratios through experiments. Our results show that DC outperforms DQ in terms of classification accuracy. The experimental results indicate that a compression ratio of 9 can be achieved with no more than 10% loss in classification accuracy.

Motor imagery classification in electroencephalography (EEG)-based brain-computer interface (BCI) systems is an important research area. To simplify the complexity of the classification, selected power bands and electrode channels have been widely used to extract and select features from raw EEG signals, but there is still a loss in classification accuracy in the state-of-the-art approaches. To solve this problem, we propose a discriminative feature extraction algorithm based on power bands with principle component analysis (PCA). First, the raw EEG signals from the motor cortex area were filtered using a bandpass filter with μ and β bands. This research considered the power bands within a 0.4 second epoch to select the optimal feature space region. Next, the total feature dimensions were reduced by PCA and transformed into a final feature vector set. The selected features were classified by applying a support vector machine (SVM). The proposed method was compared with a state-of-art power band feature and shown to improve classification accuracy.

Hardware accelerators are being increasingly deployed to boost the performance and energy efficiency of deep neural network (DNN) inference. In this paper we propose Thundervolt, a new framework that enables aggressive voltage underscaling of high-performance DNN accelerators without compromising classification accuracy even in the presence of high timing error rates. Using post-synthesis timing simulations of a DNN accelerator modeled on the Google TPU, we show that Thundervolt enables between 34%-57% energy savings on state-of-the-art speech and image recognition benchmarks with less than 1% loss in classification accuracy and no performance loss. Further, we show that Thundervolt is synergistic with and can further increase the energy efficiency of commonly used run-time DNN pruning techniques like Zero-Skip.

Support vector machines are emerging as a powerful machine-learning tool. Logarithmic number systems (LNS) utilize the property of logarithmic compression for numerical operations. We present an implementation of a digital support vector machine (SVM) classifier using LNS in which, when compared with other implementations, considerable hardware savings are achieved with no significant loss in classification accuracy.

Quantification of wheezing by a sensor system consisting of a wearable wireless acoustic sensor and smartphone performing respiratory sound classification may contribute to the diagnosis, long-term control, and lowering treatment costs of asthma. In such battery-powered sensor system, compressive sensing (CS) was verified as a method for simultaneously cutting down power cost of signal acquisition, compression, and communication on the wearable sensor. Matching real-time CS reconstruction algorithms, such as orthogonal matching pursuit (OMP), have been demonstrated on the smartphone. However, their lossy performance limits the accuracy of wheeze detection from CS-recovered short-term Fourier spectra (STFT), when using existing respiratory sound classification algorithms. Thus, here we present a novel, robust algorithm tailored specifically for wheeze detection from the CS-recovered STFT. The proposed algorithm identifies occurrence and tracks multiple individual wheeze frequency lines using hidden Markov model. The algorithm yields 89.34% of sensitivity, 96.28% of specificity, and 94.91% of accuracy on Nyquist-rate sampled respiratory sounds STFT. It enables for less than 2% loss of classification accuracy when operating over STFT reconstructed by OMP, at the signal compression ratio of up to 4 $\times$ (classification from only 25% signal samples). It features execution speed comparable to referent algorithms, and offers good prospects for parallelism.

Focused (thematic) crawling is a relatively new, promising approach to improving the recall of expert search on the Web. It involves the automatic classification of visited documents into a user- or community-specific topic hierarchy (ontology). The quality of training data for the classifier is the most critical issue and a potential bottleneck for the effectivity and scale of a focused crawler. This paper presents the BINGO! approach to focused crawling that aims to overcome the limitations of initial training data. To this end, BINGO! identifies, among the crawled and positively classified documents of a topic, characteristic and uses them for periodically re-training the classifier; this way the crawler is dynamically adapted based on the most significant documents seen so far. Two kinds of archetypes are considered: good authorities as determined by employing Kleinberg's (1999) link analysis algorithm, and documents that have been automatically classified with high confidence using a linear SVM classifier. Our approach is fully implemented in the BINGO! system, and our experiments indicate that the dynamic enhancement of training data based on archetypes extends the knowledge base of the classifier by a substantial margin without loss of classification accuracy.