Improving neural network training using dynamic learning rate schedule for PINNs and image classification

The ensemble learning method is a necessary process that provides robustness and is more accurate than the single model. The snapshot ensemble convolutional neural network (CNN) has been successful and widely used in many domains, such as image classification, fault diagnosis, and plant image classification. The advantage of the snapshot ensemble CNN is that it combines the cyclic learning rate schedule in the algorithm to snap the best model in each cycle. In this research, we proposed the dropCyclic learning rate schedule, which is a step decay to decrease the learning rate value in every learning epoch. The dropCyclic can reduce the learning rate and find the new local minimum in the subsequent cycle. We evaluated the snapshot ensemble CNN method based on three learning rate schedules: cyclic cosine annealing, max-min cyclic cosine learning rate scheduler, and dropCyclic then using three backbone CNN architectures: MobileNetV2, VGG16, and VGG19. The snapshot ensemble CNN methods were tested on three aerial image datasets: UCM, AID, and EcoCropsAID. The proposed dropCyclic learning rate schedule outperformed the other learning rate schedules on the UCM dataset and obtained high accuracy on the AID and EcoCropsAID datasets. We also compared the proposed dropCyclic learning rate schedule with other existing methods. The results show that the dropCyclic method achieved higher classification accuracy compared with other existing methods.

Image classification is typically performed with highly trained feed-forward machine learning algorithms like deep neural networks and support vector machines. The image can be treated as a time-series input when applied to the network multiple times, opening the way for recurrent neural networks to perform tasks like image classification, semantic segmentation and auto-encoding. With this approach, ultra-fast training, network optimization, and short-term memory effects allows for dynamic, low-volume datasets to be quickly learned without heavy image pre-processing or feature extraction; the main limitation being that input images need labeled output images for training, as is also true of most standard approaches. In this work, the MNIST handwritten digit dataset is used as a benchmark to evaluate metrics of a modified Echo State Network for static image classification. The image array is passed through a noise filter multiple times as the Echo State Network converges to a classification. This highly dynamic approach easily adapts to sequential image (video) tasks like object tracking and is effective with small datasets. Classification rates reach 95.3% with sample size of 10000 handwritten digits and training time of approximately 5 minutes. Progression of this research enables discrete image and time-series classification under a single algorithm, with low computing power and memory requirements.

Most popular image classification methods mainly focus on classification ability rather than recognizing new things. However, human lay emphasis on cognition first and then classification, which is closely related to human memory system. Though many memory models have been proposed, they are studied in word list whereas the reports about natural images are still limited. This paper proposes a memory model for image recognition and classification based on convolutional neural network and Bayesian decision. First the image feature is extracted by convolutional neural network and stored in binary form. Then the representation, storage and retrieval processes of visual images are modeled. The test image feature vector is matched in parallel to the studied image vectors, and the likelihood values are calculated. Finally, the odd that the test image belongs to a new class is computed based on all likelihood values. If the odd value is greater than a certain threshold, the test image is regarded as new; otherwise, the Bayesian decision rule for image classification is performed. Experimental results on Caltech-101 and Caltech-256 datasets show that the proposed method can perform well in image recognition and classification tasks. And the hit probability of the method is higher than two typical methods, SRC and ELM, at present while the false alarm rate is far lower than them.

Model optimization is a problem of great concern and challenge for developing an image classification model. In image classification, selecting the appropriate hyperparameters can substantially boost the model’s ability to learn intricate patterns and features from complex image data. Hyperparameter optimization helps to prevent overfitting by finding the right balance between complexity and generalization of a model. The ensemble genetic algorithm and convolutional neural network (EGACNN) are proposed to enhance image classification by fine-tuning hyperparameters. The convolutional neural network (CNN) model is combined with a genetic algorithm GA) using stacking based on the Modified National Institute of Standards and Technology (MNIST) dataset to enhance efficiency and prediction rate on image classification. The GA optimizes the number of layers, kernel size, learning rates, dropout rates, and batch sizes of the CNN model to improve the accuracy and performance of the model. The objective of this research is to improve the CNN-based image classification system by utilizing the advantages of ensemble learning and GA. The highest accuracy is obtained using the proposed EGACNN model which is 99.91% and the ensemble CNN and spiking neural network (CSNN) model shows an accuracy of 99.68%. The ensemble approaches like EGACNN and CSNN tends to be more effective as compared to CNN, RNN, AlexNet, ResNet, and VGG models. The hyperparameter optimization of deep learning classification models reduces human efforts and produces better prediction results. Performance comparison with existing approaches also shows the superior performance of the proposed model.

Remote Sensing (RS) image classification has recently attracted great attention for its application in different tasks, including environmental monitoring, battlefield surveillance, and geospatial object detection. The best practices for these tasks often involve transfer learning from pre-trained Convolutional Neural Networks (CNNs). A common approach in the literature is employing CNNs for feature extraction, and subsequently train classifiers exploiting such features. In this paper, we propose the adoption of transfer learning by fine-tuning pre-trained CNNs for end-to-end aerial image classification. Our approach performs feature extraction from the fine-tuned neural networks and remote sensing image classification with a Support Vector Machine (SVM) model with linear and Radial Basis Function (RBF) kernels. To tune the learning rate hyperparameter, we employ a linear decay learning rate scheduler as well as cyclical learning rates. Moreover, in order to mitigate the overfitting problem of pre-trained models, we apply label smoothing regularization. For the fine-tuning and feature extraction process, we adopt the Inception-v3 and Xception inception-based CNNs, as well the residual-based networks ResNet50 and DenseNet121. We present extensive experiments on two real-world remote sensing image datasets: AID and NWPU-RESISC45. The results show that the proposed method exhibits classification accuracy of up to 98%, outperforming other state-of-the-art methods.

Image classification is a fundamental problem in the field of computer vision and involves assigning a label to an image based on its content. In this paper, we survey both traditional machine learning-based image classification methods and deep learning-based image classification methods, specifically, we review two deep learning-based image classification methods: a convolutional neural network (CNN) method and a pre-trained CNN-based transfer learning method for image classification. We first briefly review traditional machine learning-based image classification methods and then deep learning-based image classification methods, discussing both the feature extraction methods and the classification methods used in the deep learning-based methods. We discuss several deep neural network architectures for image classification, including LeNet, AlexNet, VGGNet, GoogleNet, ResNet, and DenseNet, and finally, we conclude with a discussion on the applications of image classification and compare the different methods based on various factors both qualitatively and quantitatively, and present our experimental results, while also summarizing various methods of data augmentation, batch normalization, and regularization by dropout. We introduce techniques for practical training of deep networks, and discuss fine-tuning, pruning, and model quantization for efficient inference of trained neural networks, and finally, we introduce and discuss the runtime engines for the deployment of trained neural networks for efficient inference.

BACKGROUND: Currently, artificial intelligence in the form of artificial neural networks is being actively implemented in a number of areas of our lives, including medicine. In particular, in otorhinolaryngology, artificial neural networks are used to analyze images obtained during endoscopic examinations of patients (e.g., videolaryngoscopy) [1–3]. The interpretation of laryngoscopic images often presents significant difficulties for practicing physicians, which reduces the frequency of detection of precancerous laryngeal diseases and contributes to the increase in the number of patients with stage III–IV laryngeal cancer [4, 5]. This underscores the significance of prompt performance and accurate interpretation of the findings of endoscopic examinations of patients with laryngeal disorders. Artificial neural networks can be employed to analyze the results of videolaryngoscopy, furnishing the physician with supplementary information that can enhance diagnostic accuracy and diminish the probability of error [6, 7]. AIM: The study aims to develop and train an artificial neural network for recognizing characteristic features of laryngeal neoplasms and variants of laryngeal normality. MATERIALS AND METHODS: The study was conducted under the grant of the Moscow Center for Innovative Technologies in Healthcare (grant No. 2112-1/22) entitled “Using Neural Networks (Artificial Intelligence Algorithms) for Control and Improving the Quality of Diagnosis and Treatment of Diseases of Laryngeal and Ear Structures through Digital Technologies”.The following methods were used during the course of the study: data collection for the creation of a photobank (dataset) of medical images obtained during videolaryngoscopy; data partitioning for the formation of datasets for individual nosologies and groups of diseases; the method of consilium; analysis of the accuracy of recognition and classification of digital endoscopic images; and training of classification neural networks. Consequently, a dataset comprising 1,471 laryngeal images in digital formats (JPEG, BMP) was assembled, labelled, and uploaded for the purpose of training the artificial neural network. Of the total number of images, 410 were classified as pertaining to laryngeal formation, while 1061 were classified as variants of normality. Subsequently, the neural network was trained and tested to identify the signs of normal and laryngeal masses. RESULTS: The results of the testing of the artificial neural network indicated the formation of an inaccuracy matrix, the calculation of the value of recognition accuracy, the calculation of the quality indicators of the model performance, and the construction of the ROC curve. The developed and trained artificial neural network demonstrated an accuracy of 86% in recognizing the signs of laryngeal masses and norms. CONCLUSIONS: This study demonstrates that a trained artificial neural network can successfully distinguish between signs of normal and laryngeal masses in endoscopic photographs. With further training of the neural network and achievement of high accuracy, this technology can be used in clinical practice as an assistant in the interpretation of laryngoscopic images and early diagnosis of laryngeal masses. It can also be employed to control and improve the quality of diagnosis and treatment of diseases of the throat, nose, and ears by primary care physicians.

Artificial neural networks based on sigmoidal neurons have achieved undisputed performance in various tasks, including image recognition and classification. The search for more biologically plausible artificial neuron models led to the creation of pulsed models. The new way of encoding information is through discrete pulses, and the timing of these pulses is important for the result. This work proposes the creation of a neural network with pulsed neurons for the task of image classification. The network uses cell models more similar to those found in animal brains, communicating through spikes and relying on a stochastic component for pulse generation. It also applies STDP as an unsupervised learning rule, very similar to human learning. Experiments were run using various parameter sets to study the network's dynamics in the image classification task. The results obtained were analyzed, and their performance indicates a promising method capable of good performance on three known image databases (Caltech 101, ETH-80, and MNIST). The database 1 achieved a classification accuracy of 87%, database 2 achieved 77% on ETH-80, and database 3 achieved 86% on MNIST respectively.

In recent years Neural Networks and especially Deep Neural Networks (DNN) have seen a rise in popularity. DNNs have increased the overall performance of algorithms in applications such as image recognition and classification, 2D and 3D pose detection, natural language processing, and time series forecasting. Especially in image classification and recognition, so-called Convolutional Neural Network (CNN) have gained high interest as they can reach a high accuracy, which makes them a viable solution in cancer detection or autonomous driving. As CNNs are widely used in image tasks, different visualizations techniques have bee developed to show how their internals are working. Apart from image tasks, CNNs are applicable to other problems, e.g., time series classification, time series forecasting, or natural language processing. CNNs in those contexts behave similarly, allowing for the same visualization techniques to make them more interpretable. In this chapter, we adapt image visualization algorithms to time series problems, allowing us to build granular, intuitively interpretable feature hierarchies to make a time series forecast as understandable as an image recognition task. We do so by using our previous work on power time series forecasting using CNN Auto Encoders (AEs) and applying typical CNN visualization techniques to it. Thus, we guid computer scientists to provide better interpretable figures for a time series forecasting task to application domain experts.KeywordsVisualizationConvolutional neural networksTime series data

Objectives: The paper presents application of convolution neural network and artificial neural network for image classification problem for clothing dataset along with their performance comparison against different optimizers. The major objective of this paper is to perform image classification on fashion-mnist clothing dataset images. Methods: The methods used here are, the traditional ANN and CNN. Here image classification is performed on Fashion-mnist, clothing dataset using CNN and ANN with different optimizers. The performance of the working of ANN and CNN in classifying images from fashion-mnist dataset is compared against different optimizers namely stochastic gradient Descent, Adagrad, RMS prop and Adam optimizer. Findings: The study found that CNN worked better than ANN yielding training accuracy of 95%, 93% and testing accuracy of 91%, 89% when used with Adam and RmsProp respectively. Novelty: The novelty of this work is to present a comparative study of image classification using CNN, ANN using different optimizers, since not many studies or research articles showed the performance comparison of traditional and convolution neural networks in image classification along with different optimizers. Since the real-world scenarios of today require enormous data to be processed, CNN can fit well to diversify applications since they highly reduce the number of parameters to be trained that speeds up the training process. Moreover, to be specific on image classification problems they require the best and most prominent features to be detected and uncovered; this can be achieved using CNN since it has the concept of convolution using filters at its Core. Hence, CNN is highly recommended for such image classification applications than the traditional artificial-neural-networks because of the aforementioned reasons. Keywords: ANN (Artificial NeuralNetworks); CNN (ConvolutionNeural Networks); Optimization; SGD (Stochastic Gradient Descent); RmsProp (Root mean Square propagation); Adam (Adaptive moment estimation); AdaGrad (Adaptive Gradient)

Brain tumour classification using conventional techniques involving human intervention is prone to errors. Errors in feature extraction and/or classification may turn out to be fatal. Hence focus has shifted on automated techniques for the feature extraction and classification of brain tumour images. In the present work, a mechanism for the same has been proposed involving Artificial Neural Networks (ANN). Three categories have been incorporated in the present work, viz. normal brain, brain with benign tumour and brain with malignant tumour. In the present case, MRI images of the brain have been used to train an Artificial Neural Network which subsequently predicts the category of some new MRI data. The key challenge in designing such a system is attaining high accuracy of classification. This can be achieved by accurate feature extraction mechanism and then designing an appropriate Neural Network for training and testing. At the outset, segmentation has been employed to separate the region of the tumour (if any). Later the Discrete Wavelet Transform (DWT) is used as a preprocessing tool. The DWT helps in removing the non linearities of the image signal and smoothen it out. Since images exhibit abrupt changes in pixel values, conventional Fourier methods like the Fourier Transform prove to be unfit for image analysis. Subsequently, features like contrast, correlation, energy, homogeneity, mean, standard deviation, entropy, RMS value, variance, smoothness, kurtosis and skewness have been evaluated. Prior to training the Neural Network using the feature values, Principal Component Analysis (PCA) has been employed to find trends in the feature values. A Probabilistic Neural Network (PNN) has been designed and it has been trained using the feature values. I has been found that brain tumour MRI image are classified into normal, benign and malignant category using PNN classifier with an accuracy of 97% for the used data set which can be attributed to the efficacy of the proposed method. Key Words: Brain Tumor Classification, Machine Learning, Feature Extraction, Neural Networks, Confusion Matrix, Classification Accuracy.

Despite significant advancements in image classification, deep learning models struggle to accurately discern fine details in images, producing overly confident and imbalanced predictions for certain classes. These models typically employ feature recalibration techniques but do not account for the underlying uncertainty in predictions—particularly in complex sequential tasks like image classification. These uncertainties can significantly impact the reliability of subsequent analyses, potentially compromising accuracy across various applications. To address these limitations, we introduce the Stochastic Differential Equation Recalibration Module (SDERM), a novel approach designed to dynamically adjust the channel‐wise feature responses in convolutional neural networks. It integrates a stochastic differential equation (SDE) framework into a feature recalibration module to capture the inherent uncertainties in the data and its model predictions. To the best of our knowledge, our study is the first to explore the integration of SDE‐based feature recalibration modules in image classification. We build SDERM based on two interconnected networks—drift and diffusion network. The drift network serves as a deterministic component that approximates the predictive function of the model that systematically influences recalibrations of the predictions without considering the randomness. Concurrently, the diffusion network uses the Wiener process that captures the inherent uncertainties within the data and the network's predictions. We tested the classification accuracy of SDERM in ResNet50, ResNet101, and ResNet152 against other recalibration modules, including Squeeze‐Excitation (SE), Convolutional Block Attention Module (CBAM), Gather and Excite (GE), and Position‐Aware Recalibration Module (PARM), as well as the original Bottleneck architecture. Public image classification datasets were used, including CIFAR‐10, SVHN, FashionMNIST, and HAM10000, and their classification accuracies were evaluated using the F1 score. The proposed ResNetSDE architecture achieved state‐of‐the‐art F1 scores across four of five benchmark datasets. On Fashion‐MNIST, ResNetSDE attained an F1 score of 0.937 (CI: 0.932–0.941), outperforming all baseline recalibration methods by margins of 0.9%–1.3%. For CIFAR‐10 and CIFAR‐100, ResNetSDE achieved 0.886 (CI: 0.879–0.892) and 0.962 (CI: 0.958–0.965), respectively, surpassing ResNet‐GE and ResNet‐CBAM by 3.5% and 1.3%, respectively. ResNetSDE dominated SVHN with an F1 of 0.956 (CI: 0.953–0.958), a significant improvement over ResNet‐CBAM's 0.948 (CI: 0.945–0.951). While ResNet‐CBAM led on the class‐imbalanced HAM10000 (0.770, CI: 0.758–0.782), ResNetSDE remained competitive (0.768, CI: 0.749–0.786) since its consistent superiority—evidenced by narrow confidence intervals—validates its efficacy as a feature recalibration framework. Our experiments demonstrate that SDERM can outperform existing feature recalibration modules in image classification. The integration of SDERM to ResNet enables leveraging adaptability toward the stochasticity of each dataset at various depths of the architecture in image classification where uncertainty plays a fundamental role.

Deep neural network (DNN) models, particularly convolutional neural networks (CNNs), have demonstrated remarkable performance in biomedical image classification due to their ability to automatically learn features from large datasets. One common challenge in the preparation of large, microscopic datasets for DNN tasks is sample defocusing, potentially impairing the model performance. To handle defocusing, computational imaging, or specifically quantitative phase imaging (QPI), performs digital refocusing by using both the phase and the amplitude of the complex optical field. This leads us to investigate whether feeding the complex field into DNN would potentially address the defocusing problem as in-focus information is implicitly encoded in the complex field. In this paper, we assess the feasibility of employing neural networks to directly process full amplitude and phase data from a defocus plane without digital refocusing. Our specific focus lies in understanding the tolerance for defocus in image classification neural networks when amplitude and phase are taken as inputs. To accomplish this, we used Fourier ptychography microscopy (FPM) to acquire in-focus phase and amplitude images of two distinct object types - normal red blood cells and echinocytes. We then digitally propagate the complex field to generate progressively defocused images of the samples to serve as training and testing datasets for image classification neural networks. While the digitally defocused images contain sufficient information to recover the original in-focus images, we observed that current standard implementations of deep learning models are unable to effectively utilize the defocused field to distinguish between the two cell types. We conclude that the physical-model-based digital refocusing capability of QPI remains indispensable for overcoming defocusing issues in current standard DNN models.

Cyber-Physical System (CPS) involves the combination of physical processes with computation and communication systems. The recent advancements made in cloud computing, Wireless Sensor Network (WSN), healthcare sensors, etc. tend to develop CPS as a proficient model for healthcare applications especially, home patient care. Though several techniques have been proposed earlier related to CPS structures, only a handful of studies has focused on the design of CPS models for health care sector. So, the proposal for a dedicated CPS model for healthcare sector necessitates a significant interest to ensure data privacy. To overcome the challenges, the current research paper designs a Deep Learning-based Intrusion Detection and Image Classification for Secure CPS (DLIDIC-SCPS) model for healthcare sector. The aim of the proposed DLIDIC-SCPS model is to achieve secure image transmission and image classification process for CPS in healthcare sector. Primarily, data acquisition takes place with the help of sensors and detection of intrusions is performed using Fuzzy Deep Neural Network (FDNN) technique. Besides, Multiple Share Creation (MSC) approach is used to create several shares of medical image so as to accomplish security. Also, blockchain is employed as a distributed data storage entity to create a ledger that provides access to the client. For image classification, Inception v3 with Fuzzy Wavelet Neural Network (FWNN) is utilized that diagnose the disease from the applied medical image. Finally, Salp Swarm Algorithm (SSA) is utilized to fine tune the parameters involved in WNN model, thereby boosting its classification performance. A wide range of simulations was carried out to highlight the superiority of the proposed DLIDIC-SCPS technique. The simulation outcomes confirm that DLIDIC-SCPS approach demonstrates promising results in terms of security, privacy, and image classification outcomes over recent state-of-the-art techniques.

Image Classification is one of the important processing methods in the field of image processing. Traditional Classification algorithms, such as KNN (K-NearestNeighbor), have been unable to meet the accuracy standards of image Classification. And convolutional neural networks (CNN) are increasingly used to solve the problem of image classification. This paper briefly ces the convolutional neural network, which is constructed using the TensorFlow deep learning framework, and conducts experiments on 2000 pieces of custom data set of 10 classification obtained by crawler technology. At the same time, a comparative experiment is conducted on the traditional image classification KNN algorithm. Experimental results show that the application of convintroduolutional neural network in image classification has relatively large advantages, and the accuracy has been greatly improved.