Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

With the rapid development of artificial intelligence, deep learning is developing rapidly. As the main framework of in-depth learning, convolution neural network achieved much in semantic segmentation and image classification. Based on the low speed engineering vehicle tracking system project, this paper studies the activation function, which is an important component of convolution neural network. Firstly, it summarizes the advantages and disadvantages of four traditional activation functions such as Sigmoid, and then introduces three new activation functions: Swish, FTS, and Relu-Softplus. Finally, using the TensorFlow framework, a single new activation function and a hybrid new activation function are applied to the convolution layer of the four neural networks constructed, and the image recognition comparison is performed in the data set CIFAR-10. It is concluded that the recognition precision of mixing the new activation functions is higher than that of using a new activation function alone, and the average time is shorter and the convergence speed is faster.

This paper demonstrates image classification using deep learning. Deep learning has the inherent ability to automatically discover and extract meaningful features for specific applications. Among the popular techniques in deep learning, the convolutional neural network (CNN) stands out. CNN consists of an input layer, hidden layers, and an output layer, where meaningful features are automatically extracted from input images. This paper presents the performance and identifies the most effective CNN architectures for accurately classifying images in the CIFAR-10 dataset. Five CNN architectures were implemented, namely [Architecture 1], [Architecture 2], [Architecture 3], [Architecture 4], and [Architecture 5] using the CIFAR-10 dataset. The architectures were selected based on the need to explore variations in convolutional filter sizes, dense layers, and batch normalization to assess their impact on CIFAR-10 image classification performance. Each architecture was trained on a standard training set and evaluated on a validation set. We used specific details on data preprocessing and training settings for a consistent and fair comparison. After training and evaluation, we have obtained the following results for each architecture. Architecture 1 has a training accuracy of 74.7% and validation accuracy of 76.6%, Architecture 2 has 96.09% and 86.08%, Architecture 3 has 77% and 78.1%, Architecture 4 has 67.91% and 69.54%, and Architecture 5 has 94.64% and 87.34% of training accuracy and validation accuracy respectively. After conducting a comparative analysis, we found that Architecture 5 has achieved the highest validation accuracy in classifying images in the CIFAR-10 dataset. These findings suggest that Architecture 5 is a promising choice for image classification tasks involving the CIFAR-10 dataset.

Disease prediction based on modeling the correlations between compounded indicator factors is a widely used technique in high incidence chronic disease prevention diagnosis. Predictive models based on personal health information have been developed historically by using simple regression fitting over relatively few factors. Regression approaches have been favored in previous prediction modeling approaches because they are simplest and do not assume any non-linearity in the model for contributions of the chosen factors. In practice, many factors are correlated and have underlying non-linear relationships to the predicted outcome. Deep learning offers a means to construct a more complex modeling approach, along with automation and adaptation. The aim of this paper is to assess the ability of a deep learning model to predict the heart disease incidence using a common benchmark dataset (University of California, Irvine (UCI) dataset). The performance of deep learning model has been compared with four popular machine learning models (two linear and two nonlinear) in predicting the incidence of heart disease using data from 567 participants from two cohorts taken from UCI database. The deep learning model was able to achieve the best accuracy of 94% and an AUC score of 0.964 when compared to other models. The performance of deep learning and nonlinear machine learning models was significantly better compared to the linear machine learning models with increase in the dataset size.

This dissertation explores the critical role of loss functions in enhancing the predictive performance of deep machine learning models. Loss functions are an integral element of all the ongoing advances we witness daily in this domain. I design custom loss functions and their impacts on various machine learning tasks, particularly in computer vision. In the first stage of my research, I aim to improve the prediction performance of deep learning models by providing them with more precise feedback associated with task requirements. This led me to create the concept of assistive loss functions. My first proposed loss function, inspired by the Active Shape Model (ASM), is ASM Loss, designed to improve the detection of facial landmarks and head pose accuracy in facial images. Additionally, influenced by Knowledge Distillation (KD), I introduce KD-Loss, a loss designed specifically for regression tasks, focusing on detecting facial landmarks. Proceeding to the second phase of my research, I focus on regression problems. Building upon my prior work, I introduce Adaptive Coordinate-based Regression (ACR)-Loss, a context-aware loss function with face alignment implication. I provide a mechanism to dynamically measure how hard the localization of each landmark point is, and accordingly, ACR-Loss provides unique feedback to the model. Furthermore, I introduce Intensity-aware Categorical (IC)-Loss for detecting sagittal cervical spine landmark points in medical images. It effectively deploys characteristics from classification problems to enhance regression optimization. Both ACR-Loss and IC-Loss comprise contextual information, leading to a significant enhancement in localization precision. In the third phase, I turned my attention toward classification problems within the domain of deep machine learning. I introduce Adaptive Correlation-based (Ad-corre) Loss, designed to enhance the discriminative capabilities of deep learning models, with a specific focus on improving classification accuracy in automatic facial expression recognition. Furthermore, I introduce Informative (Info) Loss, a specialized loss function developed for medical applications, specifically aimed at distinguishing between Mild Cognitive Impairment and Normal Cognition groups based on transcriptions of interviews conducted as part of the I-CONECT study. Overall, I have proposed custom loss functions to improve the predictive performance in both regression and classification problems. Furthermore, I have introduced assistive loss functions, designed to enhance the performance of the main loss functions. The collective outcomes of this extensive exploration highlight the critical role of loss functions, adapting and optimizing them for various domains in deep machine learning. This research suggests and proposes a variety of methodologies for designing adaptive and context-aware loss functions within deep machine learning.

Emotion recognition (ER) is crucial for understanding human behaviours, social interactions, and psychological well-being. Electroencephalography (EEG) has emerged as a promising tool for capturing the neural correlates of emotions. This work is a systematic review of articles in ER using EEG signals. A total of 120 articles from 1041 articles were selected based on PRISMA guidelines using defined inclusion and exclusion criteria, published between 2018 and 2024. This article aims to provide an in-depth understanding of the current landscape of ER from EEG signals utilizing deep learning (DL). This review offers valuable guidance for researchers and practitioners seeking more refined and reliable emotion classification systems. To explore the effectiveness of DL models in EEG-based ER, several potential DL models, such as convolutional neural network, long short-term memory (LSTM), gated recurrent unit (GRU), hybrid bidirectional LSTM (BiLSTM), bidirectional GRU, and advanced DL models such as convolutional recurrent neural network and EEG-Conformer models are applied to two popular datasets, SEED and GAMEEMO, respectively, to depict the full process of ER. Additionally, the performance of DL models is also compared with the performance of basic machine learning (ML) models such as SVM, k-nearest neighbors, logistic regression, and boosting algorithms such as AdaBoost, XGBoost and LightGBM. Through extensive experiments and performance evaluations, the performance of different models when applied to the datasets mentioned above is compared. The accuracy, precision, recall, and F1-scores are analysed to determine the most effective model for EEG-based ER. The findings of this study demonstrate that the performance of hybrid DL models is more efficacious than that of ML models. The best-performing model (BiLSTM) classified the emotions, with an accuracy of 90.54% when applied to the GAMEEMO dataset. This research contributes to the growing body of literature on ER and provides insights into the feasibility of using EEG signals to understand emotional states, and presents a structured roadmap for future exploration. The findings can aid in the development of more accurate and reliable ER systems, which can have wide-ranging applications in psychology, social sciences, and human-computer interactions.

Although the existing traditional image classification methods have been widely applied in practical problems, there are some problems in the application process, such as unsatisfactory effects, low classification accuracy, and weak adaptive ability. This method separates image feature extraction and classification into two steps for classification operation. The deep learning model has a powerful learning ability, which integrates the feature extraction and classification process into a whole to complete the image classification test, which can effectively improve the image classification accuracy. However, this method has the following problems in the application process: first, it is impossible to effectively approximate the complex functions in the deep learning model. Second, the deep learning model comes with a low classifier with low accuracy. So, this paper introduces the idea of sparse representation into the architecture of the deep learning network and comprehensively utilizes the sparse representation of well multidimensional data linear decomposition ability and the deep structural advantages of multilayer nonlinear mapping to complete the complex function approximation in the deep learning model. And a sparse representation classification method based on the optimized kernel function is proposed to replace the classifier in the deep learning model, thereby improving the image classification effect. Therefore, this paper proposes an image classification algorithm based on the stacked sparse coding depth learning model-optimized kernel function nonnegative sparse representation. The experimental results show that the proposed method not only has a higher average accuracy than other mainstream methods but also can be good adapted to various image databases. Compared with other deep learning methods, it can better solve the problems of complex function approximation and poor classifier effect, thus further improving image classification accuracy.

The rapid growth of both the Industrial Internet of Things (IIoT) and Artificial Intelligence (AI) results in a high demand for AI applications in devices. To achieve high levels of accuracy, AI applications typically require a large amount of annotated data. Accessing such data is challenging in various applications such as healthcare, finance and information security. Federated learning (FL) is one of the strategies that was proposed to overcome this challenge. Specifically, FL enables the AI model in the centralized system to be trained without any prior knowledge of the information on the devices. Recent FLs have the disadvantage that they are dependent upon a centralized system, and thus are susceptible to single points of failure. This paper proposes a strategy that employs FL in a decentralized environment where devices can communicate with each other to increase the accuracy of the AI model in each device. Furthermore, we evaluate the proposed strategy in the image classification task with the ResNet50 architecture and the CIFAR-10 dataset. The evaluation shows that the ResNet50 model trained in the decentralized environment can achieve comparable results to the model trained in the centralized environment.

Area under the curve may hide poor generalisation to external datasets

This aimed at the problem of insufficient samples of training datasets for deep convolutional neural networks in computer vision tasks, which may easily lead to over-fitting, a Gaussian Adjusted Local Dynamic Transform (GA-LDT) data enhancement algorithm based on Gaussian adjustment was proposed. The distribution characteristics of the function perform a local dynamic transformation on the image, enhance the difference and diversity of image information, and alleviate the problem of overfitting. To verify the algorithm's effectiveness, comparative experiments on ResNet, Yolox, and Espnet networks suitable for different tasks are carried out on CIFAR-100, CIFAR-10, Pascal VOC, and Cityscapes data sets, respectively. After using the GA-LDT algorithm, the accuracy of the ResNet network on CIFAR-100 and CIFAR-10 data sets has increased by 3.86% and 7.61%, respectively, year-on-year. Yolox network uses GA-LDT algorithm, and the mAP on Pascal VOC dataset is increased by 1.86% year-on-year, while the mloU on Cityscapes dataset of Espnet network using GA-LDT algorithm is increased by 0.51% year-on-year.

Soil temperature (ST) plays an important role in agriculture and other fields, and has a close relationship with plant growth and development. Therefore, accurate ST prediction methods are widely needed. Deep learning (DL) models have been widely applied for ST prediction. However, the traditional DL models may fail to capture the spatiotemporal relationship due to its complex dependency under different related hydrologic variables. Hence, the DL models with Ensemble Empirical Mode Decomposition (EEMD) are proposed in this study. The proposed models can capture more complex spatiotemporal relationship after decomposing the ST into different intrinsic mode functions. Therefore, the performance of models is further improved. The results show that the performance of DL models with EEMD are better than that of corresponding DL models without EEMD. Moreover, EEMD-Conv3d has the best performance among all the experimental models. It has the highest R2 ranging from 0.9826 to 0.9893, the lowest RMSE ranging from 1.3096 to 1.6497 and the lowest MAE ranging from 0.9656 to 1.2056 in predicting ST at the lead time from one to five days. In addition, the lines between predicted ST and observed ST are closer to the ideal line (y = x) than other DL models. The results show that our EEMD-Conv3D can better capture spatiotemporal correlation and is an applicable method for predicting spatiotemporal ST.

The activation functions are essential in neural networks because they are the ones responsible for gaining the abstract characteristics of the input through the use of nonlinear transformations. The paper presents a new activation function, which is referred to as Hybrid Discontinuous Activation Function (HDAF). It is continuously differentiable except at zero, unbounded above, bounded below, and non-monotonic. Our results demonstrate that HDAF outperforms ReLU on a number of challenging datasets and neural network models. In the beginning of the experiment, neural networks are trained and classified using benchmark data like Breast Cancer Wisconsin (Diagnostic) and Iris Flower datasets and HDAF achieved 92.98% and 95.56% accuracy, respectively. Secondly, experiments were conducted on VGG16 over the MNIST and CIFAR10 datasets. The HDAF obtained 99.21%, and 84.95% accuracy respectively. Statistical feature measurements demonstrate that HDAF has the best mean accuracy, lowest standard deviation, lowest Root Mean squared error, lowest variance, and lowest Mean squared error. The study indicated that HDAF has faster convergence compared to ReLU, making it a valuable factor in deep learning. The findings from the experiments indicate that HDAF can be a promising for ReLU, leading to better performance in neural network models.

This study focuses on the optimization and practice of Convolutional Neural Networks (CNN) in image classification tasks. A deep neural network model was constructed by introducing the CBAM (Convolutional Block Attention Module) attention mechanism and residual structure. This model combines the stability of residual networks during training with the advantages of CBAM in feature extraction, aiming to improve image classification performance. The experiment used the CIFAR-10 dataset and achieved a testing accuracy of 86% through optimization methods such as data augmentation, batch normalization, reasonable control of the number of fully connected layers, and exponential decay learning rate strategy. The study also explored the effects of data augmentation, batch normalization layer utility, fully connected layer quantity control, residual structure effectiveness, and CBAM attention mechanism on model performance, providing valuable insights for optimizing CNN in image classification tasks.

This paper presents a supervised learning scheme that employs key-frame extraction to enhance the performance of pre-trained deep learning models for object detection in surveillance videos. Developing supervised deep learning models requires a significant amount of annotated video frames as training data, which demands substantial human effort for preparation. Key frames, which encompass frames containing false negative or false positive objects, can introduce diversity into the training data and contribute to model improvements. Our proposed approach focuses on detecting false negatives by leveraging the motion information within video frames that contain the detected object region. Key-frame extraction significantly reduces the human effort involved in video frame extraction. We employ interactive labeling to annotate false negative video frames with accurate bounding boxes and labels. These annotated frames are then integrated with the existing training data to create a comprehensive training dataset for subsequent training cycles. Repeating the training cycles gradually improves the object detection performance of deep learning models to monitor a new environment. Experiment results demonstrate that the proposed learning approach improves the performance of the object detection model in a new operating environment, increasing the mean average precision (mAP@0.5) from 54% to 98%. Manual annotation of key frames is reduced by 81% through the proposed key-frame extraction method.

Pathological is crucial to cancer diagnosis. Usually, Pathologists draw their conclusion based on observed cell and tissue structure on histology slides. Rapid development in machine learning, especially deep learning have established robust and accurate classifiers. They are being used to analyze histopathological slides and assist pathologists in diagnosis. Most machine learning systems rely heavily on annotated data sets to gain experiences and knowledge to correctly and accurately perform various tasks such as classification and segmentation. This work investigates different granularity of annotations in histopathological data set including image-wise, bounding box, ellipse-wise, and pixel-wise to verify the influence of annotation in pathological slide on deep learning models. We design corresponding experiments to test classification and segmentation performance of deep learning models based on annotations with different annotation granularity. In classification, state-of-the-art deep learning-based classifiers perform better when trained by pixel-wise annotation dataset. On average, precision, recall and F1-score improves by 7.87%, 8.83% and 7.85% respectively. Thus, it is suggested that finer granularity annotations are better utilized by deep learning algorithms in classification tasks. Similarly, semantic segmentation algorithms can achieve 8.33% better segmentation accuracy when trained by pixel-wise annotations. Our study shows not only that finer-grained annotation can improve the performance of deep learning models, but also help extracts more accurate phenotypic information from histopathological slides. Intelligence systems trained on granular annotations may help pathologists inspecting certain regions for better diagnosis. The compartmentalized prediction approach similar to this work may contribute to phenotype and genotype association studies.