Evaluating the performance of hyperparameters for unbiased and fair machine learning

An official announcement by the United Nations agency has confirmed an enduring deadly disease, namely coronavirus (COVID-19). While comparing to traditional methods such as blood tests and analysis, chest X-ray (CXR) images are the most important evidential techniques to find whether the patient is normal or COVID-19. In order to distinguish the COVID-19 patient among these images there is a classification method via convolution neural network (NN) techniques adapted. In this chapter, the experimental outcomes in which histogram equalization approach is utilized to process the input CXR images. To find accurate results deep learning (DL)-based convolution neural network (CNN) achieves accuracy as 50%, subsequently reshaping the input images into 224×224 along with batch size 32 undergoes training phase and it consists of 10 epochs to generate enhanced accuracy as 96.4% by tuning hyperparameters. Moreover, we applied optimization algorithm such as Adam optimizer, RMSprop optimizer and stochastic gradient descent (SGD) in which Adam and SGD achieves validation accuracy as 96.67% yields better performance in classifying images as normal and Coronavirus disease patients.

Disturbance rejection is a fundamental problem in control engineering, and there are many methods to achieve a good disturbance rejection. One of the standard methods for the disturbance rejection is to employ an integral compensator. This compensator integrates the error between the reference signal and the output, and use the integrated value with a specific coefficient as a compensating signal. In this work, we discuss the relationship between the machine learning theory and the integral compensation. We focus on single-input-single-output discrete-time time-invariant systems, and show that the integral compensation can be understood in the context of the machine learning in this case. In particular, the integral compensation is identical to the online optimization with the standard stochastic gradient descent [1]. The above idea gives two suggestions. First, the integral compensation may become faster by employing other types of stochastic gradient descent. Many algorithms of stochastic gradient descent have been proposed in the machine learning literature, and such algorithms may improve the classical integral compensation. Second, it may become possible to reject the time-invariant state-dependent disturbance. The modeling error is a typical example of such disturbance. By learning such a disturbance, the control performance for repetitive motions will be improved compared to the integral compensation. This presentation discusses the pros and cons of regarding the integral compensation as the stochastic gradient descent optimization.

目的 基于计算机的胸腔X线影像疾病检测和分类目前存在误诊率高，准确率低的问题。本文在视觉Transformer（vision Transformer，ViT）预训练模型的基础上，通过迁移学习方法，实现胸腔X线影像辅助诊断，提高诊断准确率和效率。方法 选用带有卷积神经网络（convolutional neural network，CNN）的ViT模型，其在超大规模自然图像数据集中进行了预训练；通过微调模型结构，使用预训练的ViT模型参数初始化主干网络，并迁移至胸腔X线影像数据集中再次训练，实现疾病多标签分类。结果 在IU X-Ray数据集中对ViT迁移学习前、后模型平均AUC（area under ROC curve）得分进行对比分析实验。结果表明，预训练ViT模型平均AUC得分为0.774，与不使用迁移学习相比提升了0.208。并针对模型结构和数据预处理进行了消融实验，对ViT中的注意力机制进行可视化，进一步验证了模型有效性。最后使用Chest X-Ray14和CheXpert数据集训练微调后的ViT模型，平均AUC得分为0.839和0.806，与对比方法相比分别有0.014～0.031的提升。结论 与其他方法相比，ViT模型胸腔X线影像的多标签分类精确度更高，且迁移学习可以在降低训练成本的同时提升ViT模型的分类性能和泛化性。消融实验与模型可视化表明，包含CNN结构的ViT模型能重点关注有意义的区域，高效获取胸腔X线影像的视觉特征。;Objective The chest X-ray-relevant screening and diagnostic method is essential for radiology nowadays. Most of chest X-ray images interpretation is still restricted by clinical experience and challenged for misdiagnose and missed diagnoses. To detect and identify one or more potential diseases in images automatically，it is beneficial for improving diagnostic efficiency and accuracy using computer-based technique. Compared to natural images，multiple lesions are challenged to be detected and distinguished accurately in a single image because abnormal areas have a small proportion and complex representations in chest X-ray images. Current convolutional neural network（CNN）based deep learning models have been widely used in the context of medical imaging. The structure of the CNN convolution kernel has sensitive to local detail information，and it is possible to extract richer image features. However，the convolution kernel cannot be used to get global information，and the features-extracted are restricted of redundant information like its relevance of background， muscles，and bones. The model’s performance in multi-label classification tasks are affected to a certain extent. At present，the vision Transformer（ViT）model has achieved its priorities in computer vision-related tasks. The ViT can be used to capture information simultaneously and effectively for multiple regions of the entire image. However，it is required to use large-scale dataset training to achieve good performance. Due to some factors like patient privacy and manual annotate costs，the size of the chest X-ray image data set has been limited. To reduce the model's dependence on data scale and improve the performance of multi-label classification，we develop the CNN-based ViT pre-training model in terms of the transfer learning method for diagnosis-assisted of chest X-ray image and multi-label classification. Method The CNN-based ViT model is pre-trained on a huge scale ground truth dataset，and it is used to obtain the initial parameters of the model. The model structure is fine-tuned according to the features of chest X-ray dataset. A 1×1 convolution layer is used to convert the chest X-ray images channels between 1 to 3. The number of output nodes of the linear layer in the classifier is balanced from 1 000 to the number of chest X-ray classification labels，and the Sigmoid is used as an activation function. The parameters of the backbone network are initialized in terms of the pre-trained ViT model parameters，and it is trained in the chest X-ray dataset after that to complete multi-label classification. The experiment is configured of Python3. 7 and PyTorch1. 8 to construct the model and RTX3090 GPU for training. Stochastic gradient descent（SGD）optimizer，binary cross-entropy（BCE）loss function，an initial learning rate of 1E-3，the cosine annealing learning rate decay are used. For training，each image is scaled to a size of 512×512 pixels，and a 224×224 pixels area and it is then cropped in random as the model input，and data augmentation is performed randomly by some of the flipping，perspective transformation， shearing，translation，zooming，and changing brightness. For testing，the chest X-ray image is scaled to 256×256 pixels and center crop a 224×224 area to input the trained model. Result The experiment is performed on the IU X-Ray，which is a small-scale chest X-ray dataset. This model is evaluated in quantitative using the average of area under ROC curve （AUC）scores across all classification labels. The results show that the average AUC score of the pre-trained ViT model is 0. 774. The accuracy and training efficiency of the non-pre-trained ViT model is dropped significantly. The average AUC score is reached to 0. 566 only，which is 0. 208 lower. In addition，the attention mechanism heat map is generated based on the ViT model，which can strengthen the interpretability of the model. A series of ablation experiments are carried out for data augmentation，model structure，and batch size design. The fine-tuned ViT model is trained on the Chest-Ray14 and CheXpert dataset as well. The average AUC score is reached to 0. 839 and 0. 806，which is optimized by 0. 014 and 0. 031. Conclusion A pre-trained ViT model is used for the multi-label classification of chest X-ray images via transfer learning. The experimental results illustrate that the ViT has its stronger multi-label classification performance in chest Xray images，and its attention mechanism is beneficial for lesions precision-focused like the interior of the chest cavity and the heart. Transfer learning is potential to improve the classification performance and model generalization of the ViT in small-scale datasets，and the training cost is reduced greatly. Ablation experiments demonstrate that the incorporated model of CNN and Transformer has its priority beyond single-structure model. Data enhancement and the batch size cutting can improve the performance of the model，but smaller scale of batch is still interlinked to longer training span. To improve the model's ability，we predict that future research direction can be focused on the extraction for complex disease and highlevel semantic information，such as their small lesions，disease location，and severity.

The Quran is a guideline for all Muslims. In the Quran, many things are talked about. In Quranic studies, the Quran is first classified into several topics according to the discussions of the Quranic verses. In this research, a classification model using a Back Propagation Neural Network was built based on the verses of Al-Quran and its multi-labelled topics. This allows the Back Propagation algorithm architecture to issue labels for each class in the form of `yes' or `no' for each output neuron. When using the Back Propagation algorithm, a sentence input that has become a vector is taken. In this way, TF-IDF will be used for feature extraction. Then, the model was evaluated via calculation of Hamming Loss. To ensure an optimal Back Propagation process, a comparison was made between the Stochastic Gradient Descent (SGD) and Adam optimizers. Based on some experiments, the proposed scheme yielded the best performance with a Hamming Loss value of 0.129.

SummaryFréchet inception distance (FID) has gained a better reputation as an evaluation metric for generative adversarial networks (GANs). However, it is subjected to fluctuation, namely, the same GAN model, when trained at different times can have different FID scores, due to the randomness of the weight matrices in the networks, stochastic gradient descent, and the embedded distribution (activation outputs at a hidden layer). In calculating the FIDs, embedded distribution plays the key role and it is not a trivial question from where obtaining it since it contributes to the fluctuation also. In this article, I showed that embedded distribution can be obtained from three different subspaces of the weight matrix, namely, from the row space, the null space, and the column space, and I analyzed the effect of the each space to Fréchet distances (FDs). Since the different spaces show different behaviors, choosing a subspace is not an insignificant decision. Instead of directly using the embedded distribution obtained from hidden layer's activations to calculate the FD, I proposed to use projection of embedded distribution onto the null space of the weight matrix among the three subspaces to avoid the fluctuations. My simulation results conducted at MNIST, CIFAR10, and CelebA datasets, show that, by projecting the embedded distributions onto the null spaces, possible parasitic effects coming from the randomness are being eliminated and reduces the number of needed simulationsin MNIST dataset,in CIFAR10, andin CelebA dataset.

Clickbait has been considered a problem in the modern age of technology, especially in Indonesia. Various attempts have been made to research clickbait detection, however, researches which use Indonesian data are relatively scarce compared to other languages such as English because the availability of Indonesian datasets are lacking. Backpropagation Neural Network was used in clickbait detection on article's title and has achieved quite good accuracy result, however there is still a chance to improve the accuracy. This paper shows the results of using a modified backpropagation neural network algorithm to detect clickbait using article titles when compared to the standard algorithm. The research compares the results of standard stochastic gradient descent algorithm, mini-batch gradient descent algorithm, and a version of stochastic gradient descent with Adam optimizer and three hidden layers. The results show that using Adam optimizer and three hidden layers in stochastic gradient descent algorithm significantly improves the results compared to the standard architecture. The modified algorithm shows a precision score of 78% and a recall and F1 score of 76%, where the standard algorithm has a precision score of 67% and a recall and F1 score of 66%. The resulting algorithm is then implemented to a desktop application, which is considered easy to use.

The utilization of imaging information has been accounted to be helpful for quick diagnosis of COVID-19. Although X-ray gives an assortment of indications brought about by the viral disease, given a lot of images, these visual highlights are troublesome and can set aside a long effort to be perceived by radiologists. Computerized techniques for mechanized grouping of COVID-19 on X-ray have been discovered to be promising. This paper presents an examination of the Convolutional Neural Network for the classification of COVID-19 utilizing a huge public data set of Chest X-rays gathered from COVID-19 patients and non-COVID-19 subjects. The outcomes show that utilizing various epoch counts for various optimizers such as Adaptive Moment Estimation (Adam), Stochastic Gradient Descent (SGD), and Root Mean Square Prop (RMSProp), it was obvious that APST-Net with Adam optimizer attains the highest training, validation, and F1-score of 98.45%, 98.20%, and 98.18% respectively. As binary classification was carried out, Sigmoid was chosen as the classifier and Binary-Cross Entropy is used as the loss function. The proposed model APST-Net was compared with various pre-trained models and it was concluded that the APST-Net, which is the most profound architecture, is prodigious in terms of precision, balance among affectability and particularity, and F1 score, has accomplished superior classification results. The data augmentation and manual extraction of regions of interest on the Chest X-ray images are adopted by the current implementation for COVID-19 classification.

In this study, AlexNet architecture is implemented and trained to classify C-scans featuring ancient structural patterns. The performance of two popular optimizers is examined and compared, namely the Stochastic Gradient Descent (SGD) with momentum and Adaptive Moments Estimate (Adam). The two optimizers were employed to train models using a GPR dataset from several archaeological sites. The results showed that even though SGD was more challenging to achieve learning, it eventually performed better than Adam when Batch Normalization, Dropout, and tuning the batch size and learning rate were performed. Furthermore, the generalization was tested using entirely independent data. SGD performed better, scoring 95% over 90% classification accuracy. The obtained results highlight how important the optimizer's choice can be in the learning process and is worth investigating when training CNNs models with GPR data.

Objectives: The paper presents application of convolution neural network and artificial neural network for image classification problem for clothing dataset along with their performance comparison against different optimizers. The major objective of this paper is to perform image classification on fashion-mnist clothing dataset images. Methods: The methods used here are, the traditional ANN and CNN. Here image classification is performed on Fashion-mnist, clothing dataset using CNN and ANN with different optimizers. The performance of the working of ANN and CNN in classifying images from fashion-mnist dataset is compared against different optimizers namely stochastic gradient Descent, Adagrad, RMS prop and Adam optimizer. Findings: The study found that CNN worked better than ANN yielding training accuracy of 95%, 93% and testing accuracy of 91%, 89% when used with Adam and RmsProp respectively. Novelty: The novelty of this work is to present a comparative study of image classification using CNN, ANN using different optimizers, since not many studies or research articles showed the performance comparison of traditional and convolution neural networks in image classification along with different optimizers. Since the real-world scenarios of today require enormous data to be processed, CNN can fit well to diversify applications since they highly reduce the number of parameters to be trained that speeds up the training process. Moreover, to be specific on image classification problems they require the best and most prominent features to be detected and uncovered; this can be achieved using CNN since it has the concept of convolution using filters at its Core. Hence, CNN is highly recommended for such image classification applications than the traditional artificial-neural-networks because of the aforementioned reasons. Keywords: ANN (Artificial NeuralNetworks); CNN (ConvolutionNeural Networks); Optimization; SGD (Stochastic Gradient Descent); RmsProp (Root mean Square propagation); Adam (Adaptive moment estimation); AdaGrad (Adaptive Gradient)

Medical image segmentation is crucial due to its essential role in disease therapy. Various challenges such as hair artifacts, illumination variations, and different imaging acquisitions complicate this task. In this paper, we introduce a novel convolutional neural network (CNN) architecture designed to address these challenges. We also compare our method with two well-known architectures, Unet and FCN, to demonstrate the superiority of our approach. Our results, evaluated using four metrics, accuracy, dice coefficient, Jaccard index, and sensitivity show that our method outperforms the other two. We employed Jaccard distance and binary cross-entropy as the loss functions and used SGD+Nesterov as the optimization algorithm, which proved more effective than the Adam optimizer. In the preprocessing step, we included image resizing to speed up the process and image augmentation to enhance the evaluation metrics. As a postprocessing step, we applied a threshold technique to the algorithm's outputs to increase the contrast of the final images. This method was tested on two well-known and publicly available medical image datasets: PH2 for melanoma detection and Chest X-ray images for detecting chest lesions or COVID-19.

Hyperparameter tuning is one of the essential steps to guarantee the convergence of machine learning models. We argue that intuition about the optimal choice of hyperparameters for stochastic gradient descent can be obtained by studying a neural network's phase diagram, in which each phase is characterised by distinctive dynamics of the singular values of weight matrices. Taking inspiration from disordered systems, we start from the observation that the loss landscape of a multilayer neural network with mean squared error can be interpreted as a disordered system in feature space, where the learnt features are mapped to soft spin degrees of freedom, the initial variance of the weight matrices is interpreted as the strength of the disorder, and temperature is given by the ratio of the learning rate and the batch size. As the model is trained, three phases can be identified, in which the dynamics of weight matrices is qualitatively different. Employing a Langevin equation for stochastic gradient descent, previously derived using Dyson Brownian motion, we demonstrate that the three dynamical regimes can be classified effectively, providing practical guidance for the choice of hyperparameters of the optimiser.

Tuberculosis(TBC) remains a major global health issue, with millions of new cases reported annually. Early and accurate diagnosis is essential, but manual interpretation of chest X-ray(CXR) images is limited by subjectivity and resource constrains. This study applies the MobileNetV2 architecture using transfer learning to classify tuberculosis from CXR images. The publicly available Tuberculosis Chest X-ray dataset containing 4200 images was divided into training (70%), validation (15%), and testing (15%). The pretrained MobileNetV2 model on ImageNet was used as the base network, with additional classification layers and training through the Adam optimizer and early stopping. The model achieved a validation accuracy above 99.84% after the second epoch maintained stable performance. Once the test set, model reached 99.84% accuracy, with precision 99.53% and recall 99.90% for the tuberculosis class. The result demonstrate that the transfer learning with MobileNetV2 provides a fast, efficient, and highly accurate method for tuberculosis detection. This model show potential for integration into Computer-Aided Diagnosis (CAD) system in low resource clinical settings.

Deep neural networks (DNNs) have been found useful for many applications. However, training and designing those networks can be challenging and is considered more of an art or an engineering process than rigorous science. In this regard, the important process of choosing hyperparameters is relevant. In addition, training neural networks with derivative-free methods is somewhat understudied. Particularly, with regards to hyperparameter selection. The paper presents a small-scale study of 3 hyperparameters choice for convolutional neural networks (CNNs). The networks were trained with two single-candidate optimization algorithms: Stochastic Gradient Descent (derivative-based) and Local Search (derivative-free). The CNN is trained on a subset of the FashionMNIST dataset. Experimental results show that hyperparameter selection can be detrimental for Local Search, especially regarding network parametrization. Moreover, the best hyperparameter choices didn't match for both algorithms. Future investigation into the training dynamics of Local Search is likely needed.

Federated learning over wireless networks requires aggregating locally computed gradients at a server where the mobile devices send statistically distinct gradient information over heterogenous communication links. This paper proposes a Bayesian approach for wireless federated learning referred to as soft-sign stochastic gradient descent (soft-signSGD). The idea of soft-signSGD is to aggregate the one-bit quantized local gradients at the server by jointly exploiting i) the prior distributions of the local gradients, ii) the gradient quantizer function, and iii) channel distributions. This aggregation method is optimal in the sense of minimizing the mean-squared error (MSE) under a simplified Gaussian prior assumption on the local gradient. From simulations, we demonstrate that soft-signSGD considerably outperforms the conventional sign stochastic gradient descent algorithm when training and testing neural networks using the MNIST dataset and the CIFAR-10 dataset over heterogeneous wireless networks.

Quantum machine learning (QML) has emerged as a promising field, combining the power of quantum computing with classical machine learning techniques to solve complex computational tasks. As the demand for efficient quantum simulations grows, multiple QML frameworks, including PennyLane, Qiskit, and TensorFlow Quantum (TFQ), have been developed to facilitate hybrid quantum-classical computations. This study aims to evaluate and compare the performance of three leading QML frameworks PennyLane, Qiskit, and TensorFlow Quantum in simulating quantum machine learning models, focusing on accuracy, execution time, and noise tolerance. The study examined three leading quantum machine learning (QML) frameworks PennyLane (v0.24), Qiskit (v0.43.2), and TensorFlow Quantum (TFQ, v0.7) each tested on a 64-bit Ubuntu Linux system with an Intel Core i7 processor, 32 GB RAM, and NVIDIA GeForce RTX 3080 GPU. Quantum algorithms like variational quantum circuits (VQCs) and quantum support vector machines (QSVMs) were simulated using these frameworks. Various quantum gates, including Pauli-X, Y, Z, Hadamard, CNOT, and Rotation gates (Rx, Ry, Rz), were used for parameter optimization and quantum superposition. The Iris and MNIST datasets were adapted for quantum encoding using Amplitude and Angle Encoding for binary classification tasks. Simulations were run on classical computers, while select circuits were executed on IBM Quantum’s Falcon r5.11 processor to compare hardware performance. PennyLane achieved the highest accuracy (92%) in quantum simulations, excelling in hybrid quantum-classical model integration. TensorFlow Quantum provided the fastest execution time, especially in classical simulations, making it suitable for rapid prototyping. Qiskit produced deeper quantum circuits but had higher error rates (6%) on real quantum hardware, highlighting noise and decoherence effects. The VQC models optimized by stochastic gradient descent (SGD) showed significant improvements in model performance, and QSVMs effectively classified both linearly separable and non-separable datasets, demonstrating the potential of quantum algorithms over classical approaches. Statistical analysis confirmed the significance of the results (p < 0.05). PennyLane is the most robust framework for hybrid quantum-classical models, while TensorFlow Quantum is suited for rapid prototyping. Qiskit, despite longer execution times, excels in hardware implementation. The choice of framework should align with the specific needs of quantum machine learning tasks.