Accelerating Depthwise Convolution and Pooling Operations on z-First Storage CNN Architectures

Human activity recognition involves classifying times series data, measured at inertial sensors such as accelerometers or gyroscopes, into one of pre-defined actions. Recently, convolutional neural network (CNN) has established itself as a powerful technique for human activity recognition, where convolution and pooling operations are applied along the temporal dimension of sensor signals. In most of existing work, 1D convolution operation is applied to individual univariate time series, while multi-sensors or multi-modality yield multivariate time series. 2D convolution and pooling operations are applied to multivariate time series, in order to capture local dependency along both temporal and spatial domains for uni-modal data, so that it achieves high performance with less number of parameters compared to 1D operation. However for multi-modal data existing CNNs with 2D operation handle different modalities in the same way, which cause interferences between characteristics from different modalities. In this paper, we present CNNs (CNN-pf and CNN-pff), especially CNN-pff, for multi-modal data. We employ both partial weight sharing and full weight sharing for our CNN models in such a way that modality-specific characteristics as well as common characteristics across modalities are learned from multi-modal (or multi-sensor) data and are eventually aggregated in upper layers. Experiments on benchmark datasets demonstrate the high performance of our CNN models, compared to state of the arts methods.

Convolutional neural networks (CNN) applications range from recognition and reasoning (such as handwriting recognition, facial expression recognition and video surveillance) to intelligent text applications such as semantic text analysis and natural language processing applications. Two key observations drive the design of a new architecture for CNN. First, CNN workloads exhibit a widely varying mix of three types of parallelism: parallelism within a convolution operation, intra-output parallelism where multiple input sources (features) are combined to create a single output, and inter-output parallelism where multiple, independent outputs (features) are computed simultaneously. Workloads differ significantly across different CNN applications, and across different layers of a CNN. Second, the number of processing elements in an architecture continues to scale (as per Moore's law) much faster than off-chip memory bandwidth (or pin-count) of chips. Based on these two observations, we show that for a given number of processing elements and off-chip memory bandwidth, a new CNN hardware architecture that dynamically configures the hardware on-the-fly to match the specific mix of parallelism in a given workload gives the best throughput performance. Our CNN compiler automatically translates high abstraction network specification into a parallel microprogram (a sequence of low-level VLIW instructions) that is mapped, scheduled and executed by the coprocessor. Compared to a 2.3 GHz quad-core, dual socket Intel Xeon, 1.35 GHz C870 GPU, and a 200 MHz FPGA implementation, our 120 MHz dynamically configurable architecture is 4x to 8x faster. This is the first CNN architecture to achieve real-time video stream processing (25 to 30 frames per second) on a wide range of object detection and recognition tasks.

In this study, a method of predicting the seismic responses of building structures based on a convolutional neural network (CNN) is proposed. In the method, the time histories of acceleration responses previously measured in a building during earthquakes are used in the CNN input layer, with the corresponding time histories of the displacement responses being used in the CNN output layer. The correlations between the features automatically extracted from the acceleration responses by the convolution and pooling operations in the various CNN layers and the displacement responses in the CNN output layer are established through CNN training. The trained CNN predicts the displacement responses for a future earthquake event using only the measured acceleration responses. To verify the validity of the proposed method, a numerical study on the ASCE benchmark model and an experimental study on a reinforced concrete frame structure are conducted. In the numerical study, the structural responses of the ASCE model subject to artificial earthquakes are utilized for CNN training, and the performance of the trained CNN is verified through seismic response prediction. In the experimental study, shaking table tests are performed for various earthquakes to measure the seismic responses of the reinforced concrete test structure. The seismic response prediction performance of the proposed method is examined using CNN trained with the measured seismic responses. Furthermore, the performances of CNNs trained with different numbers of datasets generated by the proposed data generation method based on data overlapping with the same data pool are discussed with related to the number of CNN training iteration number.

Mixed‐decomposed convolutional network: A lightweight yet efficient convolutional neural network for ocular disease recognition

In deep convolutional neural networks (DCNNs), model size and computation complexity are two important factors governing throughput and energy efficiency when deployed to hardware for inference. Recent works on compact DCNNs as well as pruning methods are effective, yet with drawbacks. For instance, more than half the size of all MobileNet models lies in their last two layers, mainly because compact separable convolution (CONV) layers are not applicable to their last fully connected (FC) layers. Also, in pruning methods, the compression is gained at the expense of irregularity in the DCNN architecture, which necessitates additional indexing memory to address nonzero weights, thereby increasing memory footprint, decompression delays, and energy consumption. In this article, we propose cyclic sparsely connected (CSC) architectures, with memory/computation complexity of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(N\log {}N)$ </tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> is the number of nodes/channels given a DCNN layer that, contrary to compact depthwise separable layers, can be used as an overlay for both FC and CONV layers of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(N^{2})$ </tex-math></inline-formula> . Also, contrary to pruning methods, CSC architectures are structurally sparse and require no indexing due to their cyclic nature. We show that both standard convolution and depthwise convolution layers are special cases of the CSC layers, whose mathematical function, along with FC layers, can be unified into one single formulation and whose hardware implementation can be carried out under one arithmetic logic component. We examine the efficacy of the CSC architectures for compression of LeNet, AlexNet, and MobileNet DCNNs with precision ranging from 2 to 32 bits. More specifically, we surge upon the compact 8-bit quantized 0.5 MobileNet V1 and show that by compressing its last two layers with CSC architectures, the model is compressed by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 1.5\times $ </tex-math></inline-formula> with a size of only 873 kB and little accuracy loss. Finally, we design a configurable hardware that implements all types of DCNN layers including FC, CONV, depthwise, CSC-FC, and CSC-CONV indistinguishably within a unified pipeline. We implement the hardware on a tiny Xilinx field-programmable gate array (FPGA) for total on-chip processing of the compressed MobileNet that, compared to the related work, has the highest Inference/J while utilizing the smallest FPGA.

PurposeThe aim of this study was to analyse and review deep learning convolutional neural networks for detecting and diagnosing early-stage dental caries on periapical radiographs.Materials and MethodsIn order to conduct this review, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. Studies published from 2015 to 2021 under the keywords (deep convolutional neural network) AND (caries), (deep learning caries) AND (convolutional neural network) AND (caries) were systematically reviewed.ResultsWhen dental caries is improperly diagnosed, the lesion may eventually invade the enamel, dentin, and pulp tissue, leading to loss of tooth function. Rapid and precise detection and diagnosis are vital for implementing appropriate prevention and treatment of dental caries. Radiography and intraoral images are considered to play a vital role in detecting dental caries; nevertheless, studies have shown that 20% of suspicious areas are mistakenly diagnosed as dental caries using this technique; hence, diagnosis via radiography alone without an objective assessment is inaccurate. Identifying caries with a deep convolutional neural network-based detector enables the operator to distinguish changes in the location and morphological features of dental caries lesions. Deep learning algorithms have broader and more profound layers and are continually being developed, remarkably enhancing their precision in detecting and segmenting objects.ConclusionClinical applications of deep learning convolutional neural networks in the dental field have shown significant accuracy in detecting and diagnosing dental caries, and these models hold promise in supporting dental practitioners to improve patient outcomes.

Nowadays Convolutional Neural Networks (CNNs) are being employed in a wide range of industrial technologies for a variety of sectors, such as medical, automotive, aviation, agriculture, space, etc. This paper reviews the state-of-the-art in both the field of CNNs for image classification and object detection and Autonomous Driving Systems (ADSs) in a synergetic way. Layer-based details of CNNs along with parameter and floating-point operation number calculations are outlined. Using an evolutionary approach, the majority of the outstanding image classification CNNs, published in the open literature, is introduced with a focus on their accuracy performance, parameter number, model size, and inference speed, highlighting the progressive developments in convolutional operations. Results of a novel investigation of the convolution types and operations commonly used in CNNs are presented, including a timing analysis aimed at assessing their impact on CNN performance. This extensive experimental study provides new insight into the behaviour of each convolution type in terms of training time, inference time, and layer level decomposition. Building blocks for CNN-based object detection are also discussed, such as backbone networks and baseline types, and then representative state-of-the-art designs are outlined. Experimental results from the literature are summarised for each of the reviewed models. This is followed by an overview of recent ADSs related works and current industry activities, aiming to bridge academic research and industry practice on CNNs and ADSs. Design approaches targeted at solving problems of automakers in achieving real-time implementations are also proposed based on a discussion of design constraints, human vs. machine evaluations and trade-off analysis of accuracy vs. size. Current technologies, promising directions, and expectations from the literature on ADSs are introduced including a comprehensive trade-off analysis from a human-machine perspective.

Face recognition is one of the emerging areas in the field of biometric and computer vision that plays an important role in numerous time-bound applications such as ATM payment, criminal identification, E-Learning, healthcare, and online gaming. It can be compromised by various imposter attacks such as masks, print, or replay attacks. So, there is a requirement of a light-weight powerful classifier that could take significantly less time to minimize those effects by observing the liveness of a current person. In this paper, a lightweight permuted Xceptio-Inception/Reduction Convolutional Neural Network classifier has been proposed using depthwise convolution, permutation, reshape, and residual techniques for texture-based facial liveness recognition. It has been validated with moderately dense ImageNet benchmarked Convolutional Neural Network classifiers with respect to weight size, accuracy, precision, and recall. Here, we have considered some of the variants of most popular convolution neural networks such as AlexNet, Inception, ResNet, and VGGNet and applied these models for textured based facial liveness recognition. Before the training and testing of those classifiers, all the frontal face images from the FRAUD2, NUAA, and CASIA FASD imposter datasets had normalized, and the multi-colored space LBP feature maps extracted from these normalized image frames had supplied as inputs to the classifiers. The results show that the proposed convolutional neural network performs best among the above-standardized network models, whose total weights consumes less memory space, which leads to fast liveness face recognition. In the end, comparison with the previous work shows that it achieves almost the highest success rate and lowest Equal Error Rate as a non-intrusive classifier.

The study of convolutional neural networks for 3D point clouds is becoming increasingly popular, and the difficulty lies mainly in the disorder and irregularity of point clouds. At present, it is straightforward to propose a convolution operation and perform experimental validation. Although good results are achieved, the principles behind them are not explained—i.e., why this can solve the disorder and irregularity of point clouds—and it is difficult for the researchers to design a point cloud convolution network suitable for their needs. For this phenomenon, we propose a point convolution network framework based on spatial location correspondence. Following the correspondence principle can guide us in designing convolution networks adapted to our needs. We analyzed the intrinsic mathematical nature of the convolution operation, and we argue that the convolution operation remains the same when the spatial location correspondence between the convolution kernel points and the convolution range elements remains unchanged. Guided by this principle, we formulated a general point convolution framework based on spatial location correspondence, which explains how to handle a disordered point cloud. Moreover, we discuss different kinds of correspondence based on spatial location, including M-to-M, M-to-N, and M-to-1 relationships, etc., which explain how to handle the irregularity of point clouds. Finally, we give the example of a point convolution network whose convolution kernel points are generated based on the sample’s covariance matrix distribution according to our framework. Our convolution operation can be applied to various point cloud processing networks. We demonstrated the effectiveness of our framework for point cloud classification and semantic segmentation tasks, achieving competitive results with state-of-the-art networks.

Convolutional kernel networks based on a convex combination of cosine kernels

Although deep convolutional neural networks (CNNs) have obtained outstanding performance in image super-resolution (SR), their computational cost increases geometrically as CNN models get deeper and wider. Meanwhile, the features of intermediate layers are treated equally across the channel, thus hindering the representational capability of CNNs. In this paper, we propose an attention-aware linear depthwise network for single image SR, named ALDNet. Specifically, linear depthwise convolution allows CNN-based SR models to preserve useful information for reconstructing a super-resolved image while reducing computational burden. Furthermore, we design an attention-aware branch to enhance the representation ability of depthwise convolution layers by making full use of depthwise filter interdependencies. Experiments on publicly available benchmark datasets show that ALDNet achieves superior performance to traditional depthwise separable convolutions in terms of quantitative measurements and visual quality.

This paper implements a deep learning‐based modulation pattern recognition algorithm for communication signals using a convolutional neural network architecture as a modulation recognizer. In this paper, a multiple‐parallel complex convolutional neural network architecture is proposed to meet the demand of complex baseband processing of all‐digital communication signals. The architecture learns the structured features of the real and imaginary parts of the baseband signal through parallel branches and fuses them at the output according to certain rules to obtain the final output, which realizes the fitting process to the complex numerical mapping. By comparing and analyzing several commonly used time‐frequency analysis methods, a time‐frequency analysis method that can well highlight the differences between different signal modulation patterns is selected to convert the time‐frequency map into a digital image that can be processed by a deep network. In order to fully extract the spatial and temporal characteristics of the signal, the CLP algorithm of the CNN network and LSTM network in parallel is proposed. The CNN network and LSTM network are used to extract the spatial features and temporal features of the signal, respectively, and the fusion of the two features as well as the classification is performed. Finally, the optimal model and parameters are obtained through the design of the modulation recognizer based on the convolutional neural network and the performance analysis of the convolutional neural network model. The simulation experimental results show that the improved convolutional neural network can produce certain performance gains in radio signal modulation style recognition. This promotes the application of machine learning algorithms in the field of radio signal modulation pattern recognition.

With reduced data reuse and parallelism, recent convolutional neural networks (CNNs) create new challenges for FPGA acceleration. Systolic arrays (SAs) are efficient, scalable architectures for convolutional layers, but without proper optimizations, their efficiency drops dramatically for reasons: (1) the different dimensions within same-type layers, (2) the different convolution layers especially transposed and dilated convolutions, and (3) CNN’s complex dataflow graph. Furthermore, significant overheads arise when integrating FPGAs into machine learning frameworks. Therefore, we present a flexible, composable architecture called FlexCNN, which delivers high computation efficiency by employing dynamic tiling, layer fusion, and data layout optimizations. Additionally, we implement a novel versatile SA to process normal, transposed, and dilated convolutions efficiently. FlexCNN also uses a fully pipelined software-hardware integration that alleviates the software overheads. Moreover, with an automated compilation flow, FlexCNN takes a CNN in the ONNX 1 representation, performs a design space exploration, and generates an FPGA accelerator. The framework is tested using three complex CNNs: OpenPose, U-Net, and E-Net. The architecture optimizations achieve 2.3× performance improvement. Compared to a standard SA, the versatile SA achieves close-to-ideal speedups, with up to 5.98× and 13.42× for transposed and dilated convolutions, with a 6% average area overhead. The pipelined integration leads to a 5× speedup for OpenPose.

Abstract. Deep learning models have demonstrated remarkable success in fields such as language processing and computer vision, routinely employed for tasks like language translation, image classification, and anomaly detection. Recent advancements in ocean sciences, particularly in data assimilation (DA), suggest that machine learning can emulate dynamical models, replace traditional DA steps to expedite processes, or serve as hybrid surrogate models to enhance forecasts. However, these studies often rely on ocean models of intermediate complexity, which involve significant simplifications that present challenges when transitioning to full-scale operational ocean models. This work explores the application of convolutional neural networks (CNNs) in data assimilation within the context of the HYbrid Coordinate Ocean Model (HYCOM) in the Gulf of Mexico. The CNNs are trained to correct model errors from a 2-year, high-resolution (1/25°) HYCOM dataset, assimilated using the Tendral Statistical Interpolation System (T-SIS). The CNNs are trained to replicate the increments generated by the T-SIS data assimilation package, aiming to correct model forecasts of sea surface temperature (SST) and sea surface height (SSH). The inputs to the CNNs include real satellite observations of SST from the Group for High Resolution Sea Surface Temperature (GHRSST), along-track altimeter SSH observations (ADT), the model background state (previous forecast), and the innovations (differences between observations and background). We assess the performance of the CNNs across five controlled experiments, designed to provide insights into their application in environments governed by full primitive equations, real observations, and complex topographies. The experiments focus on evaluating (1) the architecture and complexity of the CNNs, (2) the type and quantity of observations, (3) the type and number of assimilated fields, (4) the impact of training window size, and (5) the influence of coastal boundaries. Our findings reveal significant correlations between the chosen training window size – a factor not commonly examined – and the CNNs' ability to assimilate observations effectively. We also establish a clear link between the CNNs' architecture and complexity and their overall performance. This research uses artificial intelligence to enhance ocean forecasting in the Gulf of Mexico. By using convolutional neural networks, the study improves predictions of sea temperatures and heights by integrating real satellite data with existing models. Through five comprehensive experiments, the team found that the amount of training data and the design of the neural networks significantly affect accuracy. These insights pave the way for faster, more reliable ocean models, benefiting environmental monitoring and maritime operations.

Disentangled convolution for optimizing receptive field