A Novel Automatic CNN Architecture Design Approach Based on Genetic Algorithm

An attribution graph-based interpretable method for CNNs

Stochastic Schemata Exploiter (SSE), which is one of Evolutionary Computations, is designed to find the optimal solution of the function. When comparing it with Genetic Algorithm (GA), which is a population evolutionary computation, SSE has interesting features; quick convergence and smaller number of control parameters. In this study, SSE is applied for designing hyperparameters and structure of Convolutional Neural Network (CNN). The validity of the proposal algorithm is discussed for determining CNN in experiments using CIFAR-10. The results show that SSE can find the better parameters and structure of CNN than GA.

Convolutional neural networks (CNNs) have become a useful tool for a wide range of applications such as text classification. However, CNNs are not always sufficiently accurate to be useful in certain applications. The selection of activation functions within CNN architecture can affect the efficacy of the CNN. However, there is limited research regarding which activation functions are best for CNN text classification. This study tested sixteen activation functions across three text classification datasets and six CNN structures, to determine the effects of activation function on accuracy, iterations to convergence, and Positive Confidence Difference (PCD). PCD is a novel metric introduced to compare how activation functions affected a network’s classification confidence. Tables were presented to compare the performance of the activation functions across the different CNN architectures and datasets. Top performing activation functions across the different tests included the symmetrical multi-state activation function, sigmoid, penalised hyperbolic tangent, and generalised swish. An activation function’s PCD was the most consistent evaluation metric during activation function assessment, implying a close relationship between activation functions and network confidence that has yet to be explored.

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 203962, “Upscaling of Realistic Discrete Fracture Simulations Using Machine Learning,” by Nikolai Andrianov, SPE, Geological Survey of Denmark and Greenland, prepared for the 2021 SPE Reservoir Simulation Conference, Galveston, Texas, 4–6 October. The paper has not been peer reviewed. Upscaling of discrete fracture networks to continuum models such as the dual-porosity/dual-permeability (DP/DP) model is an industry-standard approach in modeling fractured reservoirs. In the complete paper, the author parametrizes the fine-scale fracture geometries and assesses the accuracy of several convolutional neural networks (CNNs) to learn the mapping between this parametrization and DP/DP model closures. The accuracy of the DP/DP results with the predicted model closures was assessed by a comparison with the corresponding fine-scale discrete fracture matrix (DFM) simulation of a two-phase flow in a realistic fracture geometry. The DP/DP results matched the DFM reference solution well. The DP/DP model also was significantly faster than DFM simulation. Introduction The goal of this study was to evaluate the effect of different CNN architectures on prediction accuracy for the DP/DP model closures and on the accuracy of DP/DP simulations in comparison with fine-scale DFM simulations. As a starting point, two CNN configurations were considered that have achieved breakthrough performance in image-classification tasks. The author adopted these architectures to the problem of learning the mapping between pixelated fracture geometries and the DP/DP model closures and indicated several key features in the CNN structure that are crucial for achieving high prediction accuracy. Mapping of fracture geometries requires significant effort, which limits the possibilities for creating large training data sets with realistic fracture geometries. The author, therefore, used the synthetic random linear fractures’ data set to train the CNNs and the fracture geometry from the Lägerdorf outcrop for testing purposes. It was demonstrated that an optimal CNN configuration yielded the DP/DP model closures such that the corresponding DP/DP results matched well the two-phase DFM simulations on a subset of the Lägerdorf data. The run times for the DP/DP model were a fraction of the time needed to accomplish the DFM simulations. Problem formulation is presented in a series of equations in the complete paper.

Upscaling of discrete fracture networks to continuum models such as the dual porosity/dual permeability (DPDP) model is an industry-standard approach in modelling of fractured reservoirs. While flow-based upscaling provides more accurate results than analytical methods, the application of flow-based upscaling is limited due to its high computational cost. In this work, we parametrize the fine-scale fracture geometries and assess the accuracy of several convolutional neural networks (CNNs) to learn the mapping between this parametrization and the DPDP model closures such as the upscaled fracture permeabilities and the matrix-fracture shape factors. We exploit certain similarities between this task and the problem of image classification and adopt several best practices from the state-of-the-art CNNs used for image classification. By running a sensitivity study, we identify several key features in the CNN structure which are crucial for achieving high accuracy of predictions for the DPDP model closures, and put forward the corresponding CNN architectures. Obtaining a suitable training dataset is challenging because i) it requires a dedicated effort to map the fracture geometries; ii) creating a conforming mesh for fine-scale simulations in presence of intersecting fractures typically leads to bad quality mesh elements; iii) fine-scale simulations are time-consuming. We alleviate some of these difficulties by pre-training a suitable CNN on a synthetic random linear fractures’ dataset and demonstrate that the upscaled parameters can be accurately predicted for a realistic fracture configuration from an outcrop data. The accuracy of the DPDP results with the predicted model closures is assessed by a comparison with the corresponding fine-scale discrete fracture-matrix (DFM) simulation of a two-phase flow in a realistic fracture geometry. The DPDP results match well the DFM reference solution, while being significantly faster than the latter.

In this work, a framework based on Convolution Neural Network (CNN) is proposed for speech emotion recognition (SER). We focus on extracting the most salient frames via the proposed CNN structure from the entire frame sequence to represent the utterance. A particular pooling method named global k-max pooling is utilized in our CNN structure (GCNN) to achieve the above objective. We implemented SER experiments on Interactive Emotional Dyadic Motion Capture (IEMOCAP), results are compared to those of some other CNN structures to validate the advancement of the presented framework. The experimental results turn out that GCNN outperforms others CNN models. Besides, experiments are also done to explore how many key frames should be output from GCNN to involve salient emotional information, results illuminate that limited length representation is properer while too long representation is likely containing redundant information decreasing the performance of the model.

Text classification is a classic task in the field of natural language processing. however, the existing methods of text classification tasks still need to be improved because of the complex abstraction of text semantic information and the strong relecvance of context. In this paper, we combine the advantages of two traditional neural network model, Long Short-Term Memory(LSTM) and Convolutional Neural Network(CNN). LSTM can effectively preserve the characteristics of historical information in long text sequences, and extract local features of text by using the structure of CNN. We proposes a hybrid model of LSTM and CNN, construct CNN model on the top of LSTM, the text feature vector output from LSTM is further extracted by CNN structure. The performance of the hybrid model is compared with that of other models in the experiment. The experimental results show that the hybrid model can effectively improve the accuracy of text classification.

In general, Convolutional Neural Networks (CNNs) have a complex network structure consisted of heavy layers with huge number of parameters such as the convolutional, pooling, relu-activation, and fully-connected layers. Due to the complexity and computation load, CNNs are trained on a cloud environment. There are a couple of drawbacks on learning and performing on the cloud such as security problem of personal information and dependency of communication state. Recently, CNNs are directly trained at the mobile devices in order to alleviate those two drawbacks. Due to the resource limitation of the mobile devices, the structure of CNNs needs to be compressed or to reduce training overhead. In this paper, we propose an on-device partial learning technique with the following benefits: (1) does not require additional neural network structures, and (2) reduces unnecessary computation overhead. We select a subset of influential weights from a trained network to accommodate the new classification class. The selection is made based on the information of the contribution of each weight to output, which is measured using the entropy concept. In the experimental section, we demonstrate and analyze our method with a CNN image classifier using two datasets such as Mixed National Institute of Standards and Technology image data and Microsoft Common Objection in Context data. As a result, the computational resources at LeNet-5 and AlexNet showed performance improvements of 1.7× and 2.3×, respectively, and memory resources demonstrated performance improvements of 1.4× and 1.6×, respectively.

Due to the irregular and sporadic nature of wildfires, continuous monitoring of large areas is required. Since geostationary satellite sensors can observe large areas with high temporal resolution, they are suitable for monitoring wildfires in real time. However, the threshold algorithm currently employed for satellite-based active fire detection has poor performance in sensors with low spatial resolution. In addition, the algorithm does not account for environmental conditions that affect wildfire detection, resulting in poor generalization performance for large areas. This study examines the viability of an adaptive active fire detection model by combining satellite and numerical model data with deep learning. A model for active fire detection was developed using commonly employed brightness temperature-related variables (key variables) and local environmental variables (sub variables). Key variables are the cross spectral and spatial differences between the MIR (central wavelength of 3.85 m) and 2 TIR (central wavelengths of 9.63 and 11.20 m) channels of the Advanced Himawari Imager (AHI). Sub variables include Solar zenith angle (SOZ) and satellite zenith angle (SAZ) of AHI, skin temperature (ST) and relative humidity (RH) of European Centre for Medium-Range Weather Forecasts (ECMWF) Reanalysis v5 (ERA5)-land data. Four processes (confidence, frequency, land cover, and continuity tests) were used to extract reference fire samples from Moderate Resolution Imaging Spectroradiometer (MODIS) and Visible Infrared Imaging Radiometer Suite (VIIRS) active fire products. To consider the different properties of key and sub variables, a 2-way convolutional neural network (CNN) structure was developed. To evaluate the influence of environmental variables, a CNN model without sub variables was adopted as a control model. The 2-way CNN (recall of 0.86, precision of 0.96, and standard deviation of recall of 0.13) was more robust at five focus sites than the control CNN (recall of 0.82, precision of 0.97, and standard deviation of recall of 0.163). Despite having a lower spatial resolution than MODIS/VIIRS, 2-way CNN outperformed other satellite-based active fire products (MODIS, VIIRS, AHI, and Advanced Meteorological Imager) in terms of detection capacity. The control CNN demonstrated poor performance under certain environmental conditions (high RH, high SAZ, and transition time between day and night), but 2-way CNN mitigates this tendency. In particular, the use of RH improved detection sensitivity, and SAZ contributed to the spatial robustness. This study demonstrated the significance of environmental conditions in active fire detection and proposed a suitable CNN structure for this intent. Based on the findings of this study, higher-level adaptive active fire monitoring under diverse environmental conditions will be possible together with explainable artificial intelligence.

Artificial Neural networks have been proved most efficient in Deep Learning mainly because of large number of datasets it can handle. The most widely used is the Convolutional Neural Network (CNN). It has been proved useful for computer vision, pattern recognition and Natural Language Processing (NLP). CNN is so vastly used, as, unlike traditional Neural Nets, it reduces number of parameters and focus more on domain specific features. There are various CNN architectures proposed, such as LeNet, AlexNet, GoogleNet. In this paper, we talk about structure of CNN and all the models of CNN which are proposed till date.

Rapid quantitative analysis of cotton-polyester blended fabrics using near-infrared spectroscopy combined with CNN-LSTM

In order to identify pests quickly and accurately in agricultural environment, a deep learning model based on convolutional neural network is proposed to locate and identify pests. In this paper, by accelerating the color attenuation of the whole image, combining with the spatial influence between the hyperpixel regions, the saliency value of each super region is calculated, and then the potential area of the pest target is provided. Finally, the pest target location and segmentation are combined with grabcut algorithm. For the segmented pest target, the optimized convolution neural network is used to express and classify, and the structure of convolution neural network is further reduced.

Self-attention based models such as vision transformers (ViTs) have emerged as a very competitive architecture alternative to convolutional neural networks (CNNs) in computer vision. Despite increasingly stronger variants with ever higher recognition accuracies, due to the quadratic complexity of self-attention, existing ViTs are typically demanding in computation and model size. Although several successful design choices (e.g., the convolutions and hierarchical multi-stage structure) of prior CNNs have been reintroduced into recent ViTs, they are still not sufficient to meet the limited resource requirements of mobile devices. This motivates a very recent attempt to develop light ViTs based on the state-of-the-art MobileNet-v2, but still leaves a performance gap behind. In this work, pushing further along this under-studied direction we introduce EdgeViTs, a new family of light-weight ViTs that, for the first time, enable attention based vision models to compete with the best light-weight CNNs in the tradeoff between accuracy and on-device efficiency. This is realized by introducing a highly cost-effective local-global-local (LGL) information exchange bottleneck based on optimal integration of self-attention and convolutions. For device-dedicated evaluation, rather than relying on inaccurate proxies like the number of FLOPs or parameters, we adopt a practical approach of focusing directly on on-device latency and, for the first time, energy efficiency. Extensive experiments on image classification, object detection and semantic segmentation validate high efficiency of our EdgeViTs when compared to the state-of-the-art efficient CNNs and ViTs in terms of accuracy-efficiency tradeoff on mobile hardware. Specifically, we show that our models are Pareto-optimal when both accuracy-latency and accuracy-energy tradeoffs are considered, achieving strict dominance over other ViTs in almost all cases and competing with the most efficient CNNs. Code is available at https://github.com/saic-fi/edgevit .

A genetic algorithm (GA), is used to optimize the many parameters of a convolutional neural network (CNN) that control the structure of the network. CNNs are used in image classification problems where it is necessary to generate feature descriptors to discern between image classes. Because of the deep representation of image data that CNNs are capable of generating, they are increasingly popular in research and industry applications. With the increasing number of use cases for CNNs, more and more time is being spent to come up with optimal CNN structures for different applications. Where one CNN might succeed at classification, another can fail. As a result, it is desirable to more easily find an optimal CNN structure to increase classification accuracy. In the proposed method, a GA is used to evolve the parameters that influence the structure of a CNN. The GA compares CNNs by training them on images of concrete containing cracks. The best CNN after several generations of the GA is then compared to the state-of-the-art CNN for crack detection. This work shows that it is possible to generalize the process of optimizing a CNN for image classification through the use of a GA.

This study discusses convolutional neural networks (CNNs) for vibration signals analysis, including applications in machining surface roughness estimation, bearing faults diagnosis, and tool wear detection. The one-dimensional CNNs (1DCNN) and two-dimensional CNNs (2DCNN) are applied for regression and classification applications using different types of inputs, e.g., raw signals, and time-frequency spectra images by short time Fourier transform. In the application of regression and the estimation of machining surface roughness, the 1DCNN is utilized and the corresponding CNN structure (hyper parameters) optimization is proposed by using uniform experimental design (UED), neural network, multiple regression, and particle swarm optimization. It demonstrates the effectiveness of the proposed approach to obtain a structure with better performance. In applications of classification, bearing faults and tool wear classification are carried out by vibration signals analysis and CNN. Finally, the experimental results are shown to demonstrate the effectiveness and performance of our approach.