Parameter reduction in convolutional neural networks with kernel transposition

Sentiment Analysis was a process for identifying whether a source of text contains certain opinions, emotions, and polarity. Twitter Sentiment Analysis was a process for identifying sentiment and polarity on tweet. Twitter Sentiment Analysis provided a way for did survey about public sentiment to product, or particular even through collections of tweet. Main problem in sentiment identifying was how to determine classification model that gave high accuracy to classifying sentiment of tweet. One of the method for classifying sentiment of tweet was Deep Learning. Convolutional Neural Network (CNN) was special type of architecture from Deep Learning that its architecture had convolution layer. Convolution layer was important for extract relevant feature from text for classifying sentiment. The objective of this research was for found out the best CNN model for classifying sentiment of tweet. By using a dataset of tweets about public opinion on the Smartfren 4G network service, we searched the best CNN model using 6 combination parameters, that is the computational eficiency method, window size, and dimension of word embedding for parameters in Word2Vec Skip-gram model, then activation function in convolution layer, dropout rate, and pool size in pooling layer for parameters in CNN. The test is done using 10-fold cross validation for each search for the best parameter value and produced the best CNN model with an accuracy value of 88,21%.

Designing efficient neural network architectures is a widely adopted approach to improve efficiency, besides compressing an existing deep neural network. A CNN (Convolutional Neural Network ) model typically consists of convolution layers, pooling layers, and fully-connected layers, where most of the computation comes from convolution layers. For example in ResNet-50, more than 99% multiply-accumulate operations (MACs) are from convolution layers. Therefore, designing efficient convolution layers is the core of building efficient CNN architectures. This chapter first describes the standard convolution layer and then describes three efficient variants of the standard convolution layer. Next, we present three representative manually design efficient CNN architectures, including SqueezeNet, MobileNets, and ShuffleNets. Finally, we describe automated methods for designing efficient CNN architectures. Take-aways Describes the standard convolution layer and then describes three efficient variants of the standard convolution layer. Presents three representative manually designed efficient CNN architectures, including SqueezeNet, MobileNets, and ShuffleNets. Describes automated methods for designing efficient CNN architectures.

Optimizing convolutional neural networks on multi-core vector accelerator

Background Risk stratification of heart failure (HF) in individuals without prevalent heart disease is crucial for early intervention and prevention. Electrocardiogram (ECG)-based artificial intelligence models have shown promise when predicting HF events at long follow-up periods. However, they are still limited by imbalanced case-control data and small labelled datasets. Moreover, the electrophysiological mechanisms underlying HF remain understudied. Purpose We compared the performance of a baseline convolutional neural network (CNN) model designed to predict HF events from the ECG at 8 different timepoints (1, 2, 3, 4, 6, 8, 10 & 12 years after the ECG acquisition), with a fine-tuned self-supervised model, pre-trained using a large amount of unlabelled ECGs. Subsequently, we investigated the genetic architecture of the prediction from the strongest model. Methods The baseline model consisted of 6 convolutional layers, an attentive pooling layer and 3 fully connected layers. The fine-tuned self-supervised model was composed of a frozen self-supervised feature extraction block, and a trainable classification block. The self-supervised block, composed of 6 convolutional layers, aimed to extract relevant features from the ECG and was trained with 637,438 publicly available ECGs. The classification block, composed of 1 convolutional layer, an attentive pooling layer and 3 fully connected layers, used a loss function designed to optimize the prediction of HF at the 8 timepoints. The models were trained and tested using 86,567 ECGs from UK Biobank (69,257 for training and 17,310 for test). Then, we performed a genome-wide association study (GWAS) on the standardized output at 8 years of the model with the highest area under the curve (AUC), with sex, age, body mass index, genetic array and the RR interval. Results The fine-tuned self-supervised model showed a better performance across all follow-up times when compared to the baseline model, increasing the AUC 7% on average (Figure 1). The GWAS revealed 27 candidate genes at 23 genome-wide significant loci (P < 5 x 10-8), 19 previously proposed for ECG traits, 9 of these for cardiac structural or functional traits (6 for HF), and 4 new (PATJ, BOLL, NATA/PTGIS and CCDC63). We found these genes were significantly enriched in cardiac conduction and regulation of cardiac contraction processes (Figure 2). Conclusions The use of features derived from a pretrained self-supervised model improved the performance of the CNN model, allowing a better prediction of HF at long follow-up periods. Also, the genetics underlying the ECG-based prediction of HF indicated candidate genes previously reported for ECG and arrhythmia phenotypes, highlighting electrophysiological mechanisms implicated in HF. Our findings support the integration of deep learning with genetic analyses to develop more accurate and clinically valuable risk prediction tools for early intervention and prevention of HF.AUC of the two CNN models Significantly enriched mechanisms

This study proposes a hybrid Kolmogorov-Arnold networks (KANs) and convolutional neural networks (CNN) to classify electrocardiogram (ECG) signal abnormalities in one lead ECG data of wearable telemedicine. The hybrid model combines CNN to extract hierarchical features from sequential data and KANs to model non-linear relationships with fewer parameters as an efficient classification. The study explores the model’s capacity to balance accuracy, computational efficiency, and memory usage as critical factors for real-time health monitoring in resource-constrained environments on the single-lead MIT-Beth Israel hospital (MIT-BIH) Supraventricular Arrhythmia database with five different class labels. For comparison, standalone CNN and KAN models were also trained on the same balanced dataset. The CNN model achieved an accuracy of 96.62%, precision of 96.81%, and recall of 96.53%. The KAN model, while computationally efficient, performed less effectively, with an accuracy of 94.15%, precision of 95.01%, and recall of 92.57%. In contrast, our hybrid KAN-CNN model outperformed both, attaining an accuracy of 97.53%, precision of 97.66%, recall of 97.40%, and a low loss of 0.0840. The study also explores the impact of quantization and compression on model performance, revealing that both CNN and Hybrid KAN-CNN models retained high accuracy post-quantization, whereas the KAN model exhibited a more significant drop in performance.

Software requirement analysis plays a vital role in Software Development Life Cycle (SDLC). Users requests are transformed into structured software requirements. It is required to know the class of requirements that each request belongs to. Manual classification of these requirements is time consuming . In this work, Convolutional Neural Network (CNN) model is proposed to classify software requirements into functional and non functional. The performance of CNN is affected by model architecture, embedding input word vectors, filter region size and number of filters. In this work, Binary particle swarm optimization (BPSO) is used to optimize the above parameters of CNN (CNN-BPSO) to improve the performance of CNN for software requirements classification. The proposed model is evaluated on PROMISE corpus data set which contains a set of functional and non-functional requirements. The experimental results of proposed CNN-BPSO model is able to provide better prediction accuracy than CNN model.

Development in Computer vision applications, most prominently in Artificial Intelligence (AI) over the past decade, has encouraged researchers to use computational algorithms. Machine Learning (ML) and Deep Learning (DL) algorithms are getting attention in the biomedical area to early diagnosis and prognosis of lethiferous diseases with higher accuracy using statistical analysis. Cervical cancer is one of the predominant gynecological infectious forms of cancer in females ecumenically, generally transpiring in less-developed nations. DL algorithms are more optimized and convenient due to the automatic feature extraction in Cervical lesion detection at an early phase. Here we analyze the convolutional neural network (CNN) models' performance using variation in channels depth comprises in the convolution layers to classify multi-class liquid-based cytology (LBC) Whole slide images (WSI). We propose three CNN models having two convolution layers with the number of channels (4,8), (8,16), and (32,64) and two pooling layers. CNN model with the highest number of channels in the convolution layers gives the leading performance with an accuracy of 96.89%, precision of 93.38%, the sensitivity of 93.75%, and F-score of 94.15%.

One of the diseases that attacks the lungs is pneumonia. Pneumonia is inflammation and fluid in the lungs making it difficult to breathe. This disease is diagnosed using X-Ray. Against the darker background of the lungs, infected tissue shows denser areas, which causes them to appear as white spots called infiltrates. In the image processing approach, pneumonia-infected X-rays can be detected using machine learning as well as deep learning. The convolutional neural network model is able to recognize images well and focus on points that are invisible to the human eye. Previous research using a convolutional neural network model with 10 convolution layers and 6 convolution layers has not achieved optimal accuracy. The aim of this research is to develop a convolutional neural network with a simpler architecture, namely two convolution layers and three convolution layers to solve the same problem, as well as examining the combination of various hyperparameter sizes and regularization techniques. We need to know which convolutional neural network architecture is better. As a result, the convolutional neural network classification model can recognize chest x-rays infected with pneumonia very well. The best classification model obtained an average accuracy of 89.743% with a three-layer convolution architecture, batch size 32, L2 regularization 0.0001, and dropout 0.2. The precision reached 94.091%, recall 86.456%, f1-score 89.601%, specificity 85.491, and error rate 10.257%. Based on the results obtained, convolutional neural network models have the potential to diagnose pneumonia and other diseases.

In deep convolutional neural networks (DCNNs), model size and computation complexity are two important factors governing throughput and energy efficiency when deployed to hardware for inference. Recent works on compact DCNNs as well as pruning methods are effective, yet with drawbacks. For instance, more than half the size of all MobileNet models lies in their last two layers, mainly because compact separable convolution (CONV) layers are not applicable to their last fully connected (FC) layers. Also, in pruning methods, the compression is gained at the expense of irregularity in the DCNN architecture, which necessitates additional indexing memory to address nonzero weights, thereby increasing memory footprint, decompression delays, and energy consumption. In this article, we propose cyclic sparsely connected (CSC) architectures, with memory/computation complexity of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(N\log {}N)$ </tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> is the number of nodes/channels given a DCNN layer that, contrary to compact depthwise separable layers, can be used as an overlay for both FC and CONV layers of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(N^{2})$ </tex-math></inline-formula> . Also, contrary to pruning methods, CSC architectures are structurally sparse and require no indexing due to their cyclic nature. We show that both standard convolution and depthwise convolution layers are special cases of the CSC layers, whose mathematical function, along with FC layers, can be unified into one single formulation and whose hardware implementation can be carried out under one arithmetic logic component. We examine the efficacy of the CSC architectures for compression of LeNet, AlexNet, and MobileNet DCNNs with precision ranging from 2 to 32 bits. More specifically, we surge upon the compact 8-bit quantized 0.5 MobileNet V1 and show that by compressing its last two layers with CSC architectures, the model is compressed by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 1.5\times $ </tex-math></inline-formula> with a size of only 873 kB and little accuracy loss. Finally, we design a configurable hardware that implements all types of DCNN layers including FC, CONV, depthwise, CSC-FC, and CSC-CONV indistinguishably within a unified pipeline. We implement the hardware on a tiny Xilinx field-programmable gate array (FPGA) for total on-chip processing of the compressed MobileNet that, compared to the related work, has the highest Inference/J while utilizing the smallest FPGA.

A well-known vegetable crop farmed all throughout the world is the potato. The possibility of a large number of diseases, however, could also impair the harvest. In order to quickly deal with the crop, it is vital for the planter to recognise the type of contamination. It had been determined that the leaves were a simple indicator of many diseases. Many Machine Learning (ML) techniques as well as Convolution Neural Network (CNN) models are superior in the literature for the detection of tomato crop diseases. CNN models rely on deep learning and neural networks, which are superior than traditional machine learning methods like decision trees and kNN. Although well before CNN creates beautiful artwork, the wide range of subjects they cover renders them operationally intensive. This painting suggests a less complex CNN model with eight convolution layer At the publicly accessible dataset PlantVillage, the recommended compact model performs better than the standard machine learning techniques and pre-trained fashions with an accuracy of 98.4%. The PlantVillage dataset includes 39 classes of different crops, including tomato diseases. These crops include apple, potato, maize, grapes, and more. In comparison to VGG16, which has an amazing accuracy of 93% 5% in pretrained patterns, k-NN has the absolute best accuracy of 94% in conventional ML approaches. After image augmentation, picture pre-processing was used to improve the suggested CNN's overall performance by having the ability to modify the brightness of the image using the knowledge of a prospective variable involving a randomized width of the image. The suggested model, which has a 98% accuracy rate, is also incredibly effective with datasets other than PlantVillage.

Handwritten digit recognition, a task in computer vision, is critical for applications such as postal automation, banking, and digitization of forms. Traditional approaches have leveraged statistical models, but the rise of deep learning, particularly Convolutional Neural Networks (CNN) and Long-Short Term Memory (LSTM), has revolutionized the field. However, a comprehensive performance comparison of CNN and LSTM architectures in the context of handwritten digit classification remains underexplored. This study aimed to address this gap by evaluating and comparing CNN and LSTM models for the classification of handwritten digits. Two machine learning models – Convolutional Neural Network (CNN) and Long-Short Term Memory (LSTM) were trained with the preprocessed handwritten digit dataset. The CNN model was designed with multiple convolutional and pooling layers, along with dropout for regularization. The LSTM model was designed with LSTM layers to capture sequential patterns in the data, followed by a dense layer for classification. The models were implemented in python, evaluated and compared based on accuracy, precision, recall and F1-score. The evaluation and comparison results indicate that CNN achieved 99.31% accuracy, 99.0% precision, 99.0% recall, and a 99.0% F1-score, while LSTM achieved 98.90% accuracy, 99.0% precision, 99.0% recall, and a 99.0% F1-score. The results demonstrated that CNN outperformed LSTM in terms of accuracy and misclassification errors, making it the optimal choice for image-based handwritten digit recognition. This finding underscores the efficiency of CNN in addressing challenges related to digit recognition, contributing to the advancement of automated digit classification systems and improving the accuracy of image-based classification tasks.

Ground short circuit faults in current transmission lines are common in the power systems. In order to prevent the power system from aggravating the accident caused by short-circuit faults of transmission lines, a novel convolutional neural network (CNN) model is constructed to identify the short-circuit fault of the transmission line in the power system. The CNN model is mainly consisted of five convolutional layers, three max-pooling layers, one concatenate layer, one dropout layer, one fully connected layer, and a Softmax classifier. This method uses a fixed time window to intercept system short-circuit fault data, extracts the deep features of these data from the training samples, and then corresponds the extracted features to labels one-to-one. Finally, in PSCAD/EMTDC, the new England 10 machine 39 nodes are taken as an example to realize the simulation. The experimental results show that the CNN model can quickly and accurately identify the short-circuit fault types, and the optimal model accuracy rate reaches 99.95%. The results of this manuscript -have positive effect on enhancing the disaster prevention capability and the operation stability of transmission lines.

The rapid advancement of edge computing and Tiny Machine Learning (TinyML) has created new opportunities for deploying intelligence in resource-constrained environments. With the growing demand for intelligent Internet of Things (IoT) devices that can efficiently process complex data in real-time, there is an urgent need for innovative optimisation techniques that overcome the limitations of IoT devices and enable accurate and efficient computations. This study investigates a novel approach to optimising Convolutional Neural Network (CNN) models for Hand Gesture Recognition (HGR) based on Electrical Impedance Tomography (EIT), which requires complex signal processing, energy efficiency, and real-time processing, by simultaneously reducing input complexity and using advanced model compression techniques. By systematically reducing and halving the input complexity of a 1D CNN from 40 to 20 Boundary Voltages (BVs) and applying an innovative compression method, we achieved remarkable model size reductions of 91.75% and 97.49% for 40 and 20 BVs EIT inputs, respectively. Additionally, the Floating-Point operations (FLOPs) are significantly reduced, by more than 99% in both cases. These reductions have been achieved with a minimal loss of accuracy, maintaining the performance of 97.22% and 94.44% for 40 and 20 BVs inputs, respectively. The most significant result is the 20 BVs compressed model. In fact, at only 8.73 kB and a remarkable 94.44% accuracy, our model demonstrates the potential of intelligent design strategies in creating ultra-lightweight, high-performance CNN-based solutions for resource-constrained devices with near-full performance capabilities specifically for the case of HGR based on EIT inputs.

Convolutional neural networks (CNNs) have demonstrated the state-of-the-art performances on automatic speech recognition. Softmax activation function for prediction and minimizing the cross-entropy loss is employed by most of the CNNs. This paper proposes a new deep architecture in which two heterogeneous classification techniques named as CNN and support vector machines (SVMs) are combined together. In this proposed model, features are learned using convolution layer and classified by SVMs. The last layer of CNN i.e. softmax layer is replaced by SVMs to efficiently deal with high dimensional features. This model should be interpreted as a special form of structured SVM and named as convolutional support vector machine (CSVM). Instead of training each component separately, the parameters of CNN and SVMs are jointly trained using frame level max-margin, sequence level max-margin, and state-level minimum Bayes risk criterion. The performance of CSVM is checked on TIMIT and Wall Street Journal datasets for phone recognition. By incorporating the features of both CNN and SVMs, CSVM improves the result by 13.33% and 2.31% over baseline CNN and segmental recurrent neural networks respectively.

Understanding the learning mechanism of convolutional neural networks in spectral analysis