WACSO: Wolf Crow Search Optimizer for Convolutional Neural Network Hyperparameter Optimization

Convolutional Neural Network (CNN) is a powerful and successful deep learning technique for a variety of computer vision and image analysis applications. Interpreting and explaining the decisions of CNNs is one of the most challengeable tasks despite its significant success in various image analysis tasks. Topological Data Analysis (TDA) is an approach that exploits algebraic invariants from topology to analyse high dimensional and noisy datasets as well as growing challenges of big data applications. Persistent homology (PH) is an algebraic topology method for measuring topological features of shapes and/or functions at different distance or similarity resolutions. This work is an attempt to investigate the algebraic properties of pretrained CNN convolutional layer filters based on random Gaussian/Uniform distribution. We shall investigate the stability and sensitivity of the condition number of CNN filters during and post the model training with focus on class discriminability of the PH features of the convolved images. We shall demonstrate a strong link between the condition number of the CNN filters and their discriminating power of the PH representation. In particular, we shall establish that if small perturbation added to the original images then feature maps with well-conditioned filters will produce similar topological features to the original image. Our investigation and findings are based on training CNN's with Digits, MNIST and CIFAR-10 datasets. Our ultimate interest in applying the results of these findings in designing appropriate CNN models for classifications of ultrasound tumor scan images. Preliminary results for these applications are encouraging.

Robust convolution kernel quantity determination based on corner radiation area adaptation

As an epitome of deep learning, convolutional neural network (CNN) has shown its advantages in solving many real-world problems. Successful CNN applications on medical prognosis and diagnosis have been achieved in recent years. Their common goal is to recognize the insights from the subtle details from medical images by building a suitable CNN model with maximum accuracy and minimum error. The CNN performance is extremely sensitive to the parameter tuning for any given network structure. To approach this concern, a novel self-tuning CNN model is proposed with a significant characteristic of having a metaheuristic-based optimizer. The most optimal set of parameters is often found via our proposed method, namely group theory and random selection-based particle swarm optimization (GTRS-PSO). The insights of symmetric essentials of model structure and parameter correlation are extracted, followed by the hierarchical partitioning of parameter space, and four operators on those partitions are designed for moving neighborhoods and formulating the swarm topology accordingly. The parameters are updated by a random selection strategy at each interval of partitions during the search process. Preliminary experiments over two radiology image datasets: breast cancer and lung cancer, are conducted for a comprehensive comparison of GTRS-PSO versus other optimization algorithms. The results show that CNN with GTRS-PSO optimizer can achieve the best performance for cancer image classifications, especially when there are symmetric components inside the data properties and model structures.

Training neural networks with many processors can reduce time-to-solution; however, it is challenging to maintain convergence and efficiency at large scales. The Kroneckerfactored Approximate Curvature (K-FAC) was recently proposed as an approximation of the Fisher Information Matrix that can be used in natural gradient optimizers. We investigate here a scalable K-FAC design and its applicability in convolutional neural network (CNN) training at scale. We study optimization techniques such as layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling to reduce training time while preserving convergence. We use residual neural networks (ResNet) applied to the CIFAR10 and ImageNet-1k datasets to evaluate the correctness and scalability of our K-FAC gradient preconditioner. With ResNet-50 on the ImageNet-1k dataset, our distributed K-FAC implementation converges to the 75.9% MLPerf baseline in 18-25% less time than does the classic stochastic gradient descent (SGD) optimizer across scales on a GPU cluster.

Convolutional Neural Networks (CNNs) have been shown to have high accuracy for classification tasks in numerous applications, which has resulted in their widespread adoption. However, the high accuracy of CNNs comes at the cost of high compute and bandwidth requirements for both classification and training. In this work we discuss an FPGA-based CNN training engine: FCTE, implemented using High-Level Synthesis (HLS), targeting the Xilinx Kintex Ultrascale XCKU115 device. Furthermore, we detail custom-precision floating-point (CPFP) cores for multiplication and addition implemented using HLS, which allows for reduced area utilization. We use these cores with our engine to train networks to demonstrate that an exponent width of 6 and mantissa width of 5 achieves accuracy comparable to single-precision floating-point for the MNIST and CIFAR-10 datasets. These results are achieved using round-to-zero for the CPFP multipliers and round-to-nearest for the CPFP adders, allowing for LUT savings of 32.6% for the multipliers and 21.7% for the adders when compared to half-precision floating-point, while using the same number of DSPs.

In this study, a method of predicting the seismic responses of building structures based on a convolutional neural network (CNN) is proposed. In the method, the time histories of acceleration responses previously measured in a building during earthquakes are used in the CNN input layer, with the corresponding time histories of the displacement responses being used in the CNN output layer. The correlations between the features automatically extracted from the acceleration responses by the convolution and pooling operations in the various CNN layers and the displacement responses in the CNN output layer are established through CNN training. The trained CNN predicts the displacement responses for a future earthquake event using only the measured acceleration responses. To verify the validity of the proposed method, a numerical study on the ASCE benchmark model and an experimental study on a reinforced concrete frame structure are conducted. In the numerical study, the structural responses of the ASCE model subject to artificial earthquakes are utilized for CNN training, and the performance of the trained CNN is verified through seismic response prediction. In the experimental study, shaking table tests are performed for various earthquakes to measure the seismic responses of the reinforced concrete test structure. The seismic response prediction performance of the proposed method is examined using CNN trained with the measured seismic responses. Furthermore, the performances of CNNs trained with different numbers of datasets generated by the proposed data generation method based on data overlapping with the same data pool are discussed with related to the number of CNN training iteration number.

Optimizing hyperparameters is crucial for enhancing the performance of deep learning (DL) models. The process of configuring optimal hyperparameters, known as hyperparameter tuning, can be performed using various methods. Traditional approaches like grid search and random search have significant limitations. In contrast, Bayesian optimization (BO) utilizes a surrogate model and an acquisition function to intelligently navigate the hyperparameter space, aiming to provide deeper insights into performance disparities between naïve and advanced methods. This study evaluates BO's efficacy compared to baseline methods such as random search, manual search, and grid search across multiple DL architectures, including multi-layer perceptron (MLP), convolutional neural network (CNN), and LeNet, applied to the Modified National Institute of Standards and Technology (MNIST) and CIFAR-10 datasets. The findings indicate that BO, employing the tree-structured parzen estimator (TPE) search method and expected improvement (EI) acquisition function, surpasses alternative methods in intricate DL architectures such as LeNet and CNN. However, grid search shows superior performance in smaller DL architectures like MLP. This study also adopts a multi-objective (MO) perspective, balancing conflicting performance objectives such as accuracy, F1 score, and model size (parameter count). This MO assessment offers a comprehensive understanding of how these performance metrics interact and influence each other, leading to more informed hyperparameter tuning decisions.

This paper presents a fast algorithmic method to train convolutional neural network (CNN) classifiers through extreme learning which has been verified on popular datasets on classification and pedestrian detection. CNN has been one of the best classifiers for images and object recognition. However, the Backpropagation (BP) algorithm, mostly used for training CNN, suffers from slow learning, local minimum, and poor generalization. To solve these problems, a novel architecture called CNN-ELM has been proposed here. Its core architecture is based on a local image (local receptive field) version of the ELM (Extreme Learning Machine) adopting random feature learning. Using MATLAB 2015a, classification experiments using the raw image data as input, show a comparable or mostly better classification performance compared to the BP trained CNN, with its training speed up to 200 times faster on MNIST, NORB, and CIFAR-10 datasets. Pedestrian detection experiments using INRIA datasets also exhibits much faster training than the BP trained CNN without sacrificing detection performance.

Recent research on fault diagnosis mainly focuses on how to improve diagnostic accuracy. For a given accuracy level, a variety of convolutional neural network architectures have been developed and available to achieve the specific accuracy level. With equivalent fault diagnosis accuracy, smaller convolutional neural network structure offers at least three advantages: (1) it can be deployed on hardware with limited memory, such as FPGAs; (2) the training of smaller convolutional neural networks can be faster under the same processor performance; (3) smaller network structures require less communication across severs during distributed training. So, in this paper, an improved convolutional neural network is constructed, and the following strategies are used to reduce network parameters and further make the convolutional neural networks with smaller structures: (1) equivalent replacement of large-scale convolution layers by multiple small-sized convolution layers; (2) avoid using a fully connected layer, and replace with a global pooling layer. Meanwhile, to ensure the model’s fault diagnosis accuracy and training effectiveness, inspired by the network in network, a 1*1 convolutional layer is inserted into a traditional convolutional layer to improve feature expression, and the batch-normalization layer is used to increase the training effectiveness.

The diagnosis and categorization of skin cancer, as well as the difference in skin textures and injuries, is a tough undertaking. Manually detecting skin lesions from dermoscopy images seems to be a difficult and cumbersome challenge. Recent advancements in the internet of things (IoT) and artificial intelligence for clinical applications have shown significant increase in precision and processing time. A lot of attention is given to deep learning models because they are effective at identifying cancer cells. The diagnosis and accuracy levels can be greatly increased by categorizing benign and malignant dermoscopy images. This work suggests an automated classification system based on a deep convolutional neural network (DCNN) in order to precisely perform multi-classification. The DCNN's structure was thoughtfully created by arranging a number of layers that are in charge of uniquely extracting different features from skin lesions. In this paper, we proposed a deep learning approach to tackle the three main tasks-deep extraction of features (task1) using transfer learning, selection of features (task2)-using metaheuristic algorithms such as Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO), and Gorilla Troop Optimization (GTO) as a feature selector, the extensive feature set is optimized, and the amount of features is reduced to within the range, and a two-level classification (task3) was proposed that are emerging in the field of skin lesion image processing. On the HAM10000 dataset, the proposed deep learning frameworks were assessed. The accuracy achieved on the dataset is 93.58 percent. The proposed method outperforms state-of-the-art (SOTA) techniques in terms of accuracy. The suggested technique is however highly scalable.

The report presents an algorithm for constructing a convolutional hierarchical neural network classifier, which is a modification of the algorithm for constructing hierarchical neural network classifiers suggested before. The original algorithm was designed to exploit intrinsic class hierarchy to build a class tree with a neural network in each node classifying groups of initial classes (in a non-terminal node) or a subset of original classes (in a terminal node). The convolutional modification utilizes convolutional neural networks instead of regular fully connected networks in order to apply the model to image classification tasks. Use of class hierarchy for image classification should reduce the number of adjusted neural network parameters compared to deep convolutional neural networks, and therefore it should reduce training and inference time. In this context the algorithm may be compared with some pruning techniques. The convolutional hierarchical neural network classifier inherits some hyperparameters of a conventional hierarchical neural network classifier, like the activation threshold and the threshold by the share of voting patterns. The goal of this study was to explore different strategies of choosing these hyperparameters. To test these strategies, we used the CIFAR-10 dataset. Also, for demonstration purposes we apply the convolutional hierarchical neural network classifier to the CIFAR-100 dataset.

WiFi fingerprinting using Convolutional Neural Networks (CNN) is one of the most promising techniques for indoor localisation due to the extraordinary performance of CNN in image classification. However, the performance of CNN is architecture dependant, and thus an architecture that works well in one case may not work in another, especially for the WiFi-based localisation problems. Most of the solutions use an existing hand-crafted architecture or a semi-automated CNN design for fingerprinting, which requires significant CNN expertise and time. Therefore, a satisfactory solution may not be guaranteed as it is challenging to design numerous possible architectures. In this work, we address this challenge by developing a framework that completely automates the CNN architecture design process. Our automated architectures based on VGG blocks have shown superior performance compared to standard architectures such as VGG-16. We further explore three different heuristics for automation: Bayesian optimisation, Hyperband, and Random Search and demonstrate their importance towards the automated CNN architecture development for WiFi fingerprinting. Experiments are conducted on real-world datasets and, a comparative study between our automated architecture and other models is presented. This work would, therefore, facilitate the CNN design for indoor localisation.

Stochastic Computing (SC) presents a low-cost and low-power alternative to conventional binary computing. In SC, continuous values are represented by stochastically generated bit streams. By performing simple hardware-friendly bit-wise operations on these streams, complex calculations can be realized very efficiently. However, the inherent randomness and approximation used in SC can result in undesirable computational errors. As Convolutional Neural Networks (CNNs) are inherently error-tolerant, SC could be embedded in them to gain higher speed and lower power without significant accuracy loss. In this paper, we propose using SC techniques to approximate multiplication operations on fixed-point weights and biases during training of CNNs. By employing such techniques, we demonstrate near state-of-the-art learning performance for the MNIST and CIFAR-10 datasets, while achieving significant resource and speed improvements when implementing the deep networks on a Field Programmable Gate Array (FPGA). For MNIST, we demonstrate that SC compared to conventional computing, will result in almost 3 times increase in learning speed with only 1.37% degradation in validation accuracy. Similarly, for CIFAR-10, training is accelerated 3.5 times with a degradation of 3.39%. We also show that our FPGA implementations of CNNs adopting stochastic multipliers consume over 17 times less power than their GPU counterparts.

Deep convolutional neural networks (CNNs) are widely used in many computer vision tasks. Since CNNs involve billions of computations, it is critical to reduce the resource /power consumption and improve parallelism. Compared with extensive researches on fixed point conversion for cost reduction, floating point customization has not been paid enough attention due to its higher cost than fixed point. This paper explores the customized floating point for both the training and inference of CNNs. 9-bit customized floating point is found sufficient for the training of ResNet-20 on CIFAR-10 dataset with less than 1% accuracy loss, which can also be applied to the inference of CNNs. With reduced bit-width, a computational unit (CU) based on Quad-Multiplier Packing is proposed to improve the resource efficiency of CNNs on FPGA. This design can save 87.5% DSP slices and 62.5% LUTs on Xilinx Kintex-7 platform compared to CU using 32-bit floating point. More CUs can be arranged on FPGA and higher throughput can be expected accordingly.

This study explores the potential of the GPBoost approach for groundwater quality assessment in comparison to three other gradient boosting-based algorithms. Three methods, random search, grid search, and Bayesian optimization were used to find the optimal values of various hyperparameters with all four-gradient boosting-based algorithms. One hundred and two samples of Entropy weighted water quality index with 14 input parameters are used for assessing groundwater quality. The calculated EWQI values for drinking range between 80.4 and 394.96 in pre-monsoon and 39.6 to 338.79 during the post-monsoon period. Moreover, spatial distribution maps displayed that the central portions of the study area fall under medium water quality. The performances of models were compared based on multiple statistical criteria, including Correlation Coefficient (CC), root mean square error (RMSE), and mean absolute error (MAE). The results reveal that the CC value by all modeling approaches is more than 0.93, suggesting a comparable performance by all methods. Results in terms of RMSE values in predicting the EWQI values suggest GPBoost (random search) model performed better than the other three models, thus suggesting a competitive performance by GPBoost in comparison to other gradient boosting-based approaches. Relative importance analysis provided by random and grid search methods highlights the significance of NO3 −, Mg2+, TDS, EC, and TH as important input parameters for predicting EWQI.