Command Recognition Using Binarized Convolutional Neural Network with Voice and Radar Sensors for Human-Vehicle Interaction

A pre-trained convolutional neural network (CNN) is a feed-forward computation perspective, which is widely used in the embedded systems requiring highly power-and-area efficiency. This paper realizes a binarized CNN which treats only binarized values (+1/-1) for the weights and the activation value. In this case, the multiplier is replaced by an XNOR circuit instead of a dedicated DSP block. Binarization for both weights and activation are more suitable for hardware implementation. However, the first convolutional layer still calculates in integer precision, since the input value is 8 bit RGB pixel, and not binarized. In this paper, we decompose the input value into maps of which each pixel is in 1-bit precision. The proposed method enables a binarized CNN to use bitwise operation in all layers, and shares a binarized convolutional circuit among all convolutional layers. We call this all binarized CNN. We compared our proposal with conventional ones. Since all binarized CNN do not require a dedicated DSP block, our proposal is smaller and 1.2 times faster than the typical CNNs, and almost maintains baseline classification accuracy. In addition, pipelined all binarized CNN achieved 1840 FPS, consumed 0.3 watts and its accuracy was 82.8%.

Deep Convolutional Neural Networks (CNNs) are widely employed in modern AI systems due to their unprecedented accuracy in object recognition and detection. However, it has been proven that the main bottleneck to improve large scale deep CNN based hardware implementation performance is massive data communication between processing units and off-chip memory. In this paper, we pave a way towards novel concept of in-memory convolver (IMC) that could implement the dominant convolution computation within main memory based on our proposed Spin Orbit Torque Magnetic Random Access Memory (SOT-MRAM) array architecture to greatly reduce data communication and thus accelerate Binary CNN (BCNN). The proposed architecture could simultaneously work as non-volatile memory and a reconfigurable in-memory logic (AND, OR) without add-on logic circuits to memory chip as in conventional logic-in-memory designs. The computed logic output could be also simply read out like a normal MRAM bit-cell using the shared memory peripheral circuits. We employ such intrinsic in-memory processing architecture to efficiently process data within memory to greatly reduce power-hungry and long distance data communication concerning state-of-the-art BCNN hardware. The hardware mapping results show that IMC can process the Binarized AlexNet on ImageNet data-set favorably with 134.27 μJ/img where ∼ 16x and 9x lower energy and area are achieved, respectively, compared to RRAM-based BCNN. Furthermore, 21.5% reduction in data movement in term of main memory accesses is observed compared to CPU/DRAM baseline.

The proposed deep learning algorithm will be integrated as a binary classifier under the umbrella of a multi-class classification tool to facilitate the automated detection of non-healthy deformities, anatomical landmarks, pathological findings, other anomalies and normal cases, by examining medical endoscopic images of GI tract. Each binary classifier is trained to detect one specific non-healthy condition. The algorithm analyzed in the present work expands the ability of detection of this tool by classifying GI tract image snapshots into two classes, depicting haemorrhage and non-haemorrhage state. The proposed algorithm is the result of the collaboration between interdisciplinary specialists on AI and Data Analysis, Computer Vision, Gastroenterologists of four University Gastroenterology Departments of Greek Medical Schools. The data used are 195 videos (177 from non-healthy cases and 18 from healthy cases) videos captured from the PillCam(R) Medronics device, originated from 195 patients, all diagnosed with different forms of angioectasia, haemorrhages and other diseases from different sites of the gastrointestinal (GI), mainly including difficult cases of diagnosis. Our AI algorithm is based on convolutional neural network (CNN) trained on annotated images at image level, using a semantic tag indicating whether the image contains angioectasia and haemorrhage traces or not. At least 22 CNN architectures were created and evaluated some of which pre-trained applying transfer learning on ImageNet data. All the CNN variations were introduced, trained to a prevalence dataset of 50%, and evaluated of unseen data. On test data, the best results were obtained from our CNN architectures which do not utilize backbone of transfer learning. Across a balanced dataset from no-healthy images and healthy images from 39 videos from different patients, identified correct diagnosis with sensitivity 90%, specificity 92%, precision 91.8%, FPR 8%, FNR 10%. Besides, we compared the performance of our best CNN algorithm versus our same goal algorithm based on HSV colorimetric lesions features extracted of pixel-level annotations, both algorithms trained and tested on the same data. It is evaluated that the CNN trained on image level annotated images, is 9% less sensitive, achieves 2.6% less precision, 1.2% less FPR, and 7% less FNR, than that based on HSV filters, extracted from on pixel-level annotated training data.

Although deep learning algrithms have outstanding performance in biometrics and been paid more and more attention, triditional features for should not be ignored. In this paper, fusion of partition local binary patterns (PLBP) and convolutional neural networks (CNNs) is investigated in three schemes. In serial fusion (SF) method, PLBP feature is extracted and reshaped as the input of CNNS. Decision fusion (DF) carries out the PLBP with nearest neighour classifer and CNNs seperatelly and weighted fuses the results. For feature fusion (FF), PLBP feature is reshaped and weighted fused with the feature map of CNNs. To examine the proposed methods, NCUT data set with 2040 images from 204 hands is augmented using PCA. The results indicate that when the PLBP and CNNs are merged in FF scheme with weights of 0.2 and 0.8, our fusion method reaches a state-of-the-art recognition rate of 99.95%.

Artificial intelligence for the assessment of bowel preparation

In order to overcome the effects of posture, illumination, expression and other factors on face recognition, this paper proposes an algorithm which is based on local binary pattern (LBP) and convolutional neural network (CNN). LBP is a texture description method which describes the local texture features of an image. It has good robustness of illumination and posture. CNN can effectively extract the spatial features of images and reduce the dimensions of features. This paper combines the advantages of LBP and CNN to improve the accuracy of face recognition. The CNN in this algorithm has four convolution layers, two max-pooling layers, one activation layer, one fully connected layer, and one output layer. In order to optimize the network structure, batch normalization layer is added after the convolution layer. We get the local binary pattern coded images and put the images as the input of the CNN and train the network. Hence, we can use the well-trained CNN for classification and identification. The Experiments on the CMU-PIE face database show that our algorithm can effectively improve the rate of face recognition.

Sleep arousal is defined as a shift from deep sleep to light sleep or complete awakening. Arousals cause sleep deprivation by fragmenting sleep, and ultimately, many health problems. Arousals can be induced by well-studied apneas and hypopneas or other sleep orders such as hypoventilation, bruxism, respiratory effort-related arousals. Thus, detection of less-studied non-apnea/hypopnea arousals is important for diagnosis and treatment of sleep disorders. Traditionally, polysomnography (PSG) test that is recording and inspecting overnight physiological signals is used for sleep studies. In this work, a novel method based on one dimensional local binary pattern (1D-LBP) and convolutional neural network (CNN) for automatic arousal detection from polysomnography recordings is proposed. 25 recordings from PhysioNet Challenge 2018 PSG dataset are used for experiments. Each signal in PSG recordings is transformed to a new signal using 1D-LBP, and then segmented using 10-s-long sliding window. The segments are fed to a CNN model formed by stacking 25 layers for classification of non-apnea/hypopnea arousal regions from non-arousal regions. Area under precision-recall curve (AUPRC) and area under receiver operating characteristic curve (AUROC) metrics are used for performance measurement. Experimental results reflect that the proposed method shows a great promise and obtains an AUPRC of 0.934 and an AUROC of 0.866.

Signature representations that are extracted by convolutional neural networks (CNN) can achieve low error rates. However, a trade-off exists between such models’ complexities and hand-crafted features’ slightly higher error rates. A novel writer-dependent (WD) recurrent binary pattern (RBP) network, and a novel signer identification CNN is proposed. RBP network is a recurrent neural network (RNN) to learn the sequential relation between binary pattern histograms over image windows. A novel histogram selection method is introduced to remove the stop-word codes. Dimensionality is reduced by more than 25% while improving the results. This work is the first to combine binary patterns and RNNs for static signature verification. Several test sets, derived from large-scale and popular databases (GPDS-960 and GPDS-Synthetic-10000) are used. Without training any global classifier, RBP network provides competitive equal error rates (EER). The proposed architectures are compared and integrated with other recent CNN models. Score-level integration of WD classifiers trained with different representations are investigated. Cross-validation tests demonstrate the EERs reduced compared to the best single classifier. A state-of-the-art EER of 1.11% is reported with a global decision threshold (0.57% EER with user-based thresholds) on GPDS-160 database.

Monitoring environmental conditions is an important application of cyber-physical systems. Typically, the monitoring is to perceive surrounding environments with battery-powered, tiny devices deployed in the field. While deep learning-based methods, especially the convolutional neural networks (CNNs), are promising approaches to enriching the functionalities offered by the tiny devices, they demand more computation and memory resources, which makes these methods difficult to be adopted on such devices. In this article, we develop a software framework, RAP , that permits the construction of the CNN designs by aggregating the existing, lightweight CNN layers, which are able to fit in the limited memory (e.g., several KBs of SRAM) on the resource-constrained devices satisfying application-specific timing constrains. RAP leverages the Python-based neural network framework Chainer to build the CNNs by mounting the C/C++ implementations of the lightweight layers, trains the built CNN models as the ordinary model-training procedure in Chainer, and generates the C version codes of the trained models. The generated programs are compiled into target machine executables for the on-device inferences. With the vigorous development of lightweight CNNs, such as binarized neural networks with binary weights and activations, RAP facilitates the model building process for the resource-constrained devices by allowing them to alter, debug, and evaluate the CNN designs over the C/C++ implementation of the lightweight CNN layers. We have prototyped the RAP framework and built two environmental monitoring applications for protecting endangered species using image- and acoustic-based monitoring methods. Our results show that the built model consumes less than 0.5 KB of SRAM for buffering the runtime data required by the model inference while achieving up to 93% of accuracy for the acoustic monitoring with less than one second of inference time on the TI 16-bit microcontroller platform.

Resistive random access memory (ReRAM) has been proven capable to efficiently perform in-situ matrix-vector computations in convolutional neural network (CNN) processing. The computations are often conducted on multi-level cell (MLC) that have limited precision and hence, show significant vulnerability to noises. The binarized neural network (BNN) is a hardware-friendly model that can dramatically reduce the computation and storage overheads. However, XNOR, which is the key operation in BNNs, cannot be directly computed in-situ in ReRAM because of its nonlinear behavior. To enable real in-situ processing of BNNs in ReRAM, we modified the BNN algorithm to enable direct computation of XNOR, POPCOUNT and POOL based on ReRAM cells. We also proposed the complementary resistive cell (CRC) design to efficiently conduct XNOR operations and optimized the pipeline design with decoupled buffer and computation stages. Our results show our scheme, namely, ReBNN, improves the system performance by $$25.36\times$$ and the energy efficiency by $$4.26\times$$ compared to conventional ReRAM based accelerator, and ensures a throughput higher than state-of-the-art BNN accelerators. The correctness of the modified algorithm is also validated.

The ever-growing parameter size and computation cost of Convolutional Neural Network (CNN) models hinder their deployment onto resource-constrained platforms. Network pruning techniques are proposed to remove the redundancy in CNN parameters and produce a sparse model. Sparse-aware accelerators are also proposed to reduce the computation cost and memory bandwidth requirements of inference by leveraging the model sparsity. The irregularity of sparse patterns, however, limits the efficiency of those designs. Researchers proposed to address this issue by creating a regular sparsity pattern through hardware-aware pruning algorithms. However, the pruning rate of these solutions is largely limited by the enforced sparsity patterns. This limitation motivates us to explore other compression methods beyond pruning. With two decoupled computation stages, we found that kernel decomposition could potentially take the processing of the sparse pattern off from the critical path of inference and achieve a high compression ratio without enforcing the sparse patterns. To exploit these advantages, we propose ESCALATE, an algorithm-hardware co-design approach based on kernel decomposition. At algorithm level, ESCALATE reorganizes the two computation stages of the decomposed convolution to enable a stream processing of the intermediate feature map. We proposed a hybrid quantization to exploit the different reuse frequency of each part of the decomposed weight. At architecture level, ESCALATE proposes a novel ‘Basis-First’ dataflow and its corresponding microarchitecture design to maximize the benefits brought by the decomposed convolution. We evaluate ESCALATE with four representative CNN models on both CIFAR-10 and ImageNet datasets and compare it against previous sparse accelerators and pruning algorithms. Results show that ESCALATE can achieve up to 325 × and 11 × compression ratio for models on CIFAR-10 and ImageNet, respectively. Comparing with previous dense and sparse accelerators, ESCALATE accelerator averagely boosts the energy efficiency by 8.3 × and 3.77 ×, and reduces the latency by 17.9 × and 2.16 ×, respectively.

In this paper, we classify digits written in mid-air using hand gestures. Impulse radio ultrawideband (IR-UWB) radar sensors are used for data acquisition, with three radar sensors placed in a triangular geometry. Conventional radar-based gesture recognition methods use whole raw data matrices or a group of features for gesture classification using convolutional neural networks (CNNs) or other machine learning algorithms. However, if the training and testing data differ in distance, orientation, hand shape, hand size, or even gesture speed or the radar setup environment, these methods become less accurate. To develop a more robust gesture recognition method, we propose not using raw data for the CNN classifier, but instead employing the hand's mid-air trajectory for classification. The hand trajectory has a stereotypical shape for a given digit, regardless of the hand's orientation or speed, making its classification easy and robust. Our proposed method consists of three stages: signal preprocessing, hand motion localization, and tracking and transforming the trajectory data into an image to classify it using a CNN. Our proposed method outperforms conventional approaches because it is robust to changes in orientation, distance, and hand shape and size. Moreover, this method does not require building a huge training database of digits drawn by different users in different orientations; rather, we can use training databases already available in the image processing field. Overall, the proposed mid-air handwritten digit recognition system provides a user-friendly and accurate mid-air handwriting modality that does not place restrictions on users.

The presented paper proposes a new speech command recognition model for novel engineering applications with limited resources. We built the proposed model with the help of a Convolutional Recurrent Neural Network (CRNN). The use of CRNN instead of Convolutional Neural Network (CNN) helps us to reduce the model parameters and memory requirement as per resource constraints. Furthermore, we insert transmute and curtailment layer between the layers of CRNN. By doing this we further reduce model parameters and float number of operations to half of the CRNN requirement. The proposed model is tested on Google’s speech command dataset. The obtained result shows that the proposed CRNN model requires 1/3 parameters as compared to the CNN model. The number of parameters of the CRNN model is further reduced by 45% and the float numbers of operations between 2% to 12 % in different recognition tasks. The recognition accuracy of the proposed model is 96% on Google’s speech command dataset, and on laboratory recording, its recognition accuracy is 89%.

As a popular deep learning technique, convolutional neural network has been widely used in many tasks such as image classification and object recognition. Convolutional neural network exploits spatial correlations in the images by performing convolution operations in local receptive fields. Convolutional neural networks are preferred over fully connected neural networks because they have fewer weights and are easier to train. Many research works have been conducted to reduce the computational complexity and memory requirements of convolutional neural network, to make it applicable to the low-power embedded applications with limited memories. This paper presents the architecture design of convolutional neural network with binary weights and activations, also known as binary neural network, on an FPGA platform. Weights and input activations are binarized with only two values, +1 and -1. This reduces all the fixed point multiplication operations in convolutional layers and fully connected layers to 1-bit XNOR operations. The proposed design uses only on-chip memories. Furthermore, an efficient implementation of batch normalization operation is introduced. When evaluating the CIFAR-10 benchmark, the proposed FPGA design can achieve a processing rate of 332,158 images per second with with accuracy of 86.06% using 1-bit quantized weights and activations.

Binary convolutional neural networks (BCNN) have shown good accuracy for small to medium neural network models. Their extreme quantization of weights and activations reduces off-chip data transfer and greatly reduces the computational complexity of convolutions. Further reduction in the complexity of a BCNN model for fast execution can be achieved with model size reduction at the cost of network accuracy. In this paper, a multi-model inference technique is proposed to reduce the execution time of the binarized inference process without accuracy reduction. The technique considers a cascade of neural network models with different computation/accuracy ratios. A parameterizable binarized neural network with different trade-offs between complexity and accuracy is used to obtain multiple network models. We also propose a hardware accelerator to run multi-model inference throughput in embedded systems. The multi-model inference accelerator is demonstrated on low-density Zynq-7010 and Zynq-7020 FPGA devices, classifying images from the CIFAR-10 dataset. The proposed accelerator improves the frame rate per number of LUTs by 7.2× those of previous solutions on a ZYNQ7020 FPGA with similar accuracy. This shows the effectiveness of the multi-model inference technique and the efficiency of the proposed hardware accelerator.