Optimizing Deep Learning Models with Hybrid Nonlinear Loss Functions: Integrating Heterogeneous Nonlinear Properties for Enhanced Performance

This dissertation explores the critical role of loss functions in enhancing the predictive performance of deep machine learning models. Loss functions are an integral element of all the ongoing advances we witness daily in this domain. I design custom loss functions and their impacts on various machine learning tasks, particularly in computer vision. In the first stage of my research, I aim to improve the prediction performance of deep learning models by providing them with more precise feedback associated with task requirements. This led me to create the concept of assistive loss functions. My first proposed loss function, inspired by the Active Shape Model (ASM), is ASM Loss, designed to improve the detection of facial landmarks and head pose accuracy in facial images. Additionally, influenced by Knowledge Distillation (KD), I introduce KD-Loss, a loss designed specifically for regression tasks, focusing on detecting facial landmarks. Proceeding to the second phase of my research, I focus on regression problems. Building upon my prior work, I introduce Adaptive Coordinate-based Regression (ACR)-Loss, a context-aware loss function with face alignment implication. I provide a mechanism to dynamically measure how hard the localization of each landmark point is, and accordingly, ACR-Loss provides unique feedback to the model. Furthermore, I introduce Intensity-aware Categorical (IC)-Loss for detecting sagittal cervical spine landmark points in medical images. It effectively deploys characteristics from classification problems to enhance regression optimization. Both ACR-Loss and IC-Loss comprise contextual information, leading to a significant enhancement in localization precision. In the third phase, I turned my attention toward classification problems within the domain of deep machine learning. I introduce Adaptive Correlation-based (Ad-corre) Loss, designed to enhance the discriminative capabilities of deep learning models, with a specific focus on improving classification accuracy in automatic facial expression recognition. Furthermore, I introduce Informative (Info) Loss, a specialized loss function developed for medical applications, specifically aimed at distinguishing between Mild Cognitive Impairment and Normal Cognition groups based on transcriptions of interviews conducted as part of the I-CONECT study. Overall, I have proposed custom loss functions to improve the predictive performance in both regression and classification problems. Furthermore, I have introduced assistive loss functions, designed to enhance the performance of the main loss functions. The collective outcomes of this extensive exploration highlight the critical role of loss functions, adapting and optimizing them for various domains in deep machine learning. This research suggests and proposes a variety of methodologies for designing adaptive and context-aware loss functions within deep machine learning.

Sea turtles exhibit high migratory rates and occupy a broad range of habitats, which in turn makes monitoring these taxa challenging. Applying deep learning (DL) models to vast image datasets collected from citizen science programs can offer promising solutions to overcome the challenge of monitoring the wide habitats of wildlife, particularly sea turtles. Among DL models, object detection models, such as the You Only Look Once (YOLO) series, have been extensively employed for wildlife classification. Despite their successful application in this domain, detecting objects in images with complex backgrounds, including underwater environments, remains a significant challenge. Recently, instance segmentation models have been developed to address this issue by providing more accurate classification of complex images compared to traditional object detection models. This study compared the performance of two state-of-the-art DL methods namely; the object detection model (YOLOv5) and instance segmentation model (YOLOv5-seg), to detect and classify sea turtles. The images were collected from iNaturalist and Google and then divided into 64% for training, 16% for validation, and 20% for test sets. Model performance during and after finishing training was evaluated by loss functions and various indexes, respectively. Based on loss functions, YOLOv5-seg demonstrated a lower error rate in detecting rather than classifying sea turtles than the YOLOv5. According to mean Average Precision (mAP) values, which reflect precision and recall, the YOLOv5-seg model showed superior performance than YOLOv5. The mAP0.5 and mAP0.5:0.95 for the YOLOv5 model were 0.885 and 0.795, respectively, whereas for the YOLOv5-seg, these values were 0.918 and 0.831, respectively. In particular, based on the loss functions and classification results, the YOLOv5-seg showed improved performance for detecting rather than classifying sea turtles compared to the YOLOv5. The results of this study may help improve sea turtle monitoring in the future.

Fault detection is an essential component of seismic interpretation and plays a crucial role in industrial processes. However, it is also one of the main challenges, especially in delineating faults in 3D seismic data. Recently, the rapidly developing technology, deep learning, has proven to be a powerful tool for this task. A number of neural networks have been proposed for this purpose by regarding 3D fault detection as a semantic segmentation task. To further enhance the effectiveness of the deep learning methods, we propose a novel network architecture, named R2SE-Unet, to solve the 3D segmentation problem. In the neural network, we design a recurrent residual-SE convolution unit (RRCU-SE) that integrates the residual learning and Squeeze-Excitation module to store information in 3D seismic data. This component promotes the spread of 3D volumetric information and aids in learning spatial dependencies in 3D images. In addition, to reduce the impact of insufficient spatial resolution resulting from the base architecture of U-net, we add an attention unit between skip connection operations. These two new units enable our R2SE-Unet to exploit semantic information more accurately in the feature maps. After many experiments on region-based loss functions and distribution-based loss functions, we also propose a novel loss function, which takes the advantage of generalized dice (GDice) loss and balanced binary cross entropy (b-BCE) loss, named Gdice-bce, to effectively train R2SE-Unet. Although only synthetic seismic data samples are used to train the network parameters, our R2SE-Unet could produce more reliable fault feature maps on field seismic data than two other conventional fault detection neural networks. Thus, the proposed neural network is easy to train and reliably works for seismic fault interpretation on field seismic data.

Corrosion is a natural phenomenon that causes significant material and economic losses. Timely inspection of corroded assets is vital for preventing such losses. Visual inspection is a non-destructive method that is commonly used for corrosion detection. The ability of deep learning methods to capture corrosion visual information allows for the automation of the manual inspection process. The literature includes several examples of deep learning models to detect metal corrosion. In developing such models, loss functions are crucial and directly influence the models’ performance. Although cross-entropy is a commonly used objective function for corrosion detection, systematic evaluation of alternative loss functions—and the performance gains they may unlock—has been largely neglected. This study uses eight loss functions, namely, cross-entropy loss, Focal loss, Unified Focal loss, Dice loss, Tversky loss, Focal Tversky loss, Combo loss, and Hybrid Focal loss, to train UNet-based models for corrosion segmentation and compares their performance. The results demonstrated that the model trained with the Dice loss achieved higher performance, with a mean Jaccard index value of 64.28%, while the Focal loss exhibited the lowest performance, with a mean Jaccard value of 57.20%. This study contributes to academia and practice by demonstrating the performance gains that can be achieved by tweaking the deep learning models using optimal loss functions. • Corrosion is a phenomenon that considerably impacts the environment and economy. • Deep learning models can be trained for corrosion detection, and loss functions are essential to developing accurate models. • This study evaluates the use of different loss functions, including the Focal loss, Unified Focal loss, Dice loss, Tversky loss, Focal Tversky loss, Combo loss, and Hybrid Focal loss, to train UNet-based models for corrosion segmentation. • On a standard dataset, the models developed with Combo loss, Tversky loss, Unified Focal loss, and Dice loss had higher performance than those developed with cross-entropy loss. • With the hyperparameter tuning, higher performance gains can be achieved.

The Pascal Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection. This paper describes the dataset and evaluation procedure. We review the state-of-the-art in evaluated methods for both classification and detection, analyse whether the methods are statistically different, what they are learning from the images (e.g. the object or its context), and what the methods find easy or confuse. The paper concludes with lessons learnt in the three year history of the challenge, and proposes directions for future improvement and extension.

Abstract. The deep learning (DL) models require timely updates to continue their reliability and robustness in prediction, classification, and segmentation tasks. When the deep learning models are tested with a limited test set, the model will not reveal the drawbacks. Every deep learning baseline model needs timely updates by incorporating more data, change in architecture, and hyper parameter tuning. This work focuses on updating the Conditional Generative Adversarial Network (C-GAN) based epiphyte identification deep learning model by incorporating 4 different generator architectures of GAN and two different loss functions. The four generator architectures used in this task are Resnet-6. Resnet-9, Resnet-50 and Resnet-101. A new annotation method called background removed annotation was tested to analyse the improvement in the epiphyte identification protocol. All the results obtained from the model by changing the above parameters are reported using two common evaluation metrics. Based on the parameter tuning experiment, Resnet-6, and Resnet- 9, with binary cross-entropy (BCE) as the loss function, attained higher scores also Resnet-6 with MSE as loss function performed well. The new annotation by removing the background had minimal effect on identifying the epiphytes.

Linear feature extraction is crucial for special objects in semantic segmentation networks, such as slot marking and lanes. The objects with linear characteristics have global contextual information dependency. It is very difficult to capture the complete information of these objects in semantic segmentation tasks. To improve the linear feature extraction ability of the semantic segmentation network, we propose introducing the dilated convolution with vertical and horizontal kernels (DVH) into the task of feature extraction in semantic segmentation networks. Meanwhile, we figure out the outcome if we put the different vertical and horizontal kernels on different places in the semantic segmentation networks. Our networks are trained on the basis of the SS dataset, the TuSimple lane dataset and the Massachusetts Roads dataset. These datasets consist of slot marking, lanes, and road images. The research results show that our method improves the accuracy of the slot marking segmentation of the SS dataset by 2%. Compared with other state-of-the-art methods, our UnetDVH-Linear (v1) obtains better accuracy on the TuSimple Benchmark Lane Detection Challenge with a value of 97.53%. To prove the generalization of our models, road segmentation experiments were performed on aerial images. Without data argumentation, the segmentation accuracy of our model on the Massachusetts roads dataset is 95.3%. Moreover, our models perform better than other models when training with the same loss function and experimental settings. The experiment result shows that the dilated convolution with vertical and horizontal kernels will enhance the neural network on linear feature extraction.

Our work proposes neural network design choices that set the state-of-the-art on a challenging public benchmark on cataract surgery, CaDIS. Our methodology achieves strong performance across three semantic segmentation tasks with increasingly granular surgical tool class sets by effectively handling class imbalance, an inherent challenge in any surgical video. We consider and evaluate two conceptually simple data oversampling methods as well as different loss functions. We show significant performance gains across network architectures and tasks especially on the rarest tool classes, thereby presenting an approach for achieving high performance when imbalanced granular datasets are considered. Our code and trained models are available at https://github.com/RViMLab/MICCAI2021_Cataract_semantic_segmentation and qualitative results on unseen surgical video can be found at https://youtu.be/twVIPUj1WZM.

In irrigation agriculture, predicting groundwater level (GWL) using deep learning models can help decision-makers coordinate surface water and groundwater usage, thus aiding in the sustainable development and utilization of groundwater. However, when making a long sequence prediction, prediction sequences often have severe delays affecting the availability of prediction results. In this paper, a new loss function is proposed to minimize the lag and oversmoothing on the prediction of GWLs. GWL, meteorology, and pumping data are collected via an irrigation Internet of Things system in Hutubi County, Xinjiang. Through Pearson's correlation analysis, historical potential evapotranspiration (ET0), groundwater extraction, and GWL were chosen to predict GWLs. Datasets were constructed through the proposed spatiotemporal data fusion method; then, the best model from the six deep learning models was selected by comparing the prediction capability of the datasets. Finally, the mean-squared error (MSE) loss function is replaced by the proposed loss function. Compared to the mean absolute error, MSE, and predicted sequence graphs, the new loss function significantly depresses the time delay with similar prediction accuracy.

The impact analysis of deep learning models for COVID-19-infected X-ray images is an extremely challenging task. Every model has unique capabilities that can provide suitable solutions for some given problem. The prescribed work analyzes various deep learning models that are used for capturing the chest X-ray images. Their performance-defining factors, such as accuracy, f1-score, training and the validation loss, are tested with the support of the training dataset. These deep learning models are multi-layered architectures. These parameters fluctuate based on the behavior of these layers, learning rate, training efficiency, or over-fitting of models. This may in turn introduce sudden changes in the values of training accuracy, testing accuracy, loss or validation loss, f1-score, etc. Some models produce linear responses with respect to the training and testing data, such as Xception, but most of the models provide a variation of these parameters either in the accuracy or the loss functions. The prescribed work performs detailed experimental analysis of deep learning image neural network models and compares them with the above said parameters with detailed analysis of these parameters with their responses regarding accuracy and loss functions. This work also analyses the suitability of these model based on the various parameters, such as the accuracy and loss functions to various applications. This prescribed work also lists out various challenges on the implementation and experimentation of these models. Solutions are provided for enhancing the performance of these deep learning models. The deep learning models that are used in the prescribed work are Resnet, VGG16, Resnet with VGG, Inception V3, Xception with transfer learning, and CNN. The model is trained with more than 1500 images of the chest-X-ray data and tested with around 132 samples of the X-ray image dataset. The prescribed work analyzes the accuracy, f1-score, recall, and precision of these models and analyzes these parameters. It also measures parameters such as training accuracy, testing accuracy, loss, and validation loss. Each epoch of every model is recorded to measure the changes in these parameters during the experimental analysis. The prescribed work provides insight for future research through various challenges and research findings with future directions.

The Internet of Things (IoT) holds transformative potential in fields such as power grid optimization, defense networks, and healthcare. However, the constrained processing capacities and resource limitations of IoT networks make them especially susceptible to cyber threats. This study addresses the problem of detecting intrusions in IoT environments by evaluating the performance of deep learning (DL) models under different data and algorithmic conditions. We conducted a comparative analysis of three widely used DL models—Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Bidirectional LSTM (biLSTM)—across four benchmark IoT intrusion detection datasets: BoTIoT, CiCIoT, ToNIoT, and WUSTL-IIoT-2021. Each model was assessed under balanced and imbalanced dataset configurations and evaluated using three loss functions (cross-entropy, focal loss, and dual focal loss). By analyzing model efficacy across these datasets, we highlight the importance of generalizability and adaptability to varied data characteristics that are essential for real-world applications. The results demonstrate that the CNN trained using the cross-entropy loss function consistently outperforms the other models, particularly on balanced datasets. On the other hand, LSTM and biLSTM show strong potential in temporal modeling, but their performance is highly dependent on the characteristics of the dataset. By analyzing the performance of multiple DL models under diverse datasets, this research provides actionable insights for developing secure, interpretable IoT systems that can meet the challenges of designing a secure IoT system.

The accuracy of deep learning classifiers trained using the cross entropy loss function suffers even when a fraction of training labels are wrong or input images are uninformative. Training images and labels for computational pathology are often noisy due to the difficulty in signal localization and certain disease classifications being subjective discretization of the underlying continuums of disease conditions. For training classifiers robust to input and label noise, we propose a modified and sample-specific version of generalized cross entropy loss. We take advantage of the bootstrapping properties in deep learning models in order to design loss functions that are aware of the difficulty of classifying individual samples, due to either the label noise or the lack of a strong visual signal. We carry out extensive experiments to validate our approach by comparing against the models trained using other loss functions. The superior performance of our methodology in all the experiments support the requirement of using robust loss functions for histology image classification.

The goal of this study was to see how well the multi-annotator segmentation framework performed for hippocampal head and body segmentation. In order to execute organ segmentation separately on the input MR image, the multi-annotator segmentation framework employs a number of distinct deep learning-based models with diverse loss functions, algorithms, and architectures. Each deep learning model generates a probability map, which is then fused by decision fusion algorithms or a deep learning-based combiner to produce a single binary map of the target organ/structure. The purpose of the multi-annotator segmentation method is to make use of the complimentary information generated/provided by the several independent annotators (independent deep learning models). This approach would profit from the synergistic impact of integrating the different annotators' decisions to achieve a better performance than each annotator working alone. For the delineation of the hippocampus head and body from MR images, the performance of the suggested multi-annotator approach was examined. Other segmentation approaches were used for evaluation, including a multi-view deep learning-based technique, Atlas-based methods, shape-based averaging (SBA), STAPLE, and majority voting. A variety of deep learning methods (including several architectures and loss functions) were implemented, with a residual architecture (ResNet-CE) with dilated convolutional kernels and a cross-entropy loss function demon-strating higher performance as a standalone model. For the body and head of the hippocampus, the ResNet-CE model attained Dice indices of 88 ±61.7 and 88 ±71.5, respectively. Overall, the suggested multi-annotator segmentation strategy beat other segmentation approaches with Dice indices of 91 ±01.3 for the body and 91 ±11.3 for the head, containing six separate deep learning models, including the ResNet model. Dice indices of 88 ±41.5 for the body and 88 ±51.5 for the head were obtained using the best atlas-based approach, and Dice indices of 88 ±91.5 (body) and 89 ±01.4 (head) were obtained using the multi-view segmentation method (head). This research showed that the proposed multi-annotator technique for seminal segmentation outperforms each of the annotators individually (independent deep learning models which are included in the multi-annotator approach). The proposed method could be used to improve the overall accuracy and performance of machine learning and/or deep learning-based approaches in seminal segmentation challenges.

Groundwater depth has complex non-linear relationships with climate, groundwater extraction, and surface water flows. To understand the importance of each predictor and predictand (groundwater depth), different artificial intelligence (AI) techniques have been used. In this research, we have proposed a Deep Learning (DL) model to predict groundwater depths. The DL model is an extension of the conventional neural network with multiple layers having non-linear activation function. The feasibility of the DL model is assessed with well-established framework models [Extreme Learning Machine (ELM) and Gaussian Process Regression (GPR)]. The area selected for this study is Konan basin located in the Kochi Prefecture of Japan. The hydro-meteorological and groundwater data used are precipitation, river stage, temperature, recharge and groundwater depth. Identical set of inputs and outputs of all the selected stations were used to train and validate the models. The predictive accuracy of the DL, ELM and GPR models has been assessed considering suitable goodness-of-fit criteria. During training period, the DL model has a very good agreement with the observed data (RMSE = 0.04, r = 0.99 and NSE = 0.98) and during validation period, its performance is satisfactory (RMSE = 0.08, r = 0.95 and NSE = 0.87). To check practicality and generalization ability of the DL model, it was re-validated at three different stations (E2, E3 and E6) of the same unconfined aquifer. The significant prediction capability and generalization ability makes the proposed DL model more reliable and robust. Based on the finding of this research, the DL model is an intelligent tool for predicting groundwater depths. Such advanced AI technique can save resources and labor conventionally employed to estimate various features of complex groundwater systems.

Integrating eco-biogeographical knowledge into deep learning: An approach for fine-grained mountain vegetation classification