Nonlinear multi-model ensemble prediction using dynamic Neural Network with incremental learning

The Internet of Things (IoT) has become an integral part of everyday life, and plays a significant role in various sectors by enabling device-to-device communication without human intervention. However, the constant connectivity of these devices to the Internet exposes them to numerous cyberattacks, potentially leading to data breaches, device malfunctions, and unauthorized network access. The diverse range of threats targeting IoT systems highlights the need for robust security solutions, such as machine learning-based intrusion detection systems, which have attracted growing research interest. This study proposes a novel adaptive approach for intrusion detection in IoT environments by combining batch and incremental learning models. Our work overcomes the limitations of traditional machine learning-based intrusion detection methods, emphasizing the potential of incremental learning to develop more effective and resilient solutions. Unlike most existing approaches that rely solely on batch learning and lack adaptability to integrate new data for long-term security, our method offers a more flexible alternative. Specifically, we leverage the predictive power of LightGBM in batch mode to improve the performance of a Passive-Aggressive classifier as a meta-learner in incremental mode. This dual learning approach aims to enhance intrusion detection in two key ways: effectively capturing diverse attack patterns with batch learning and adapting to emerging attack patterns with incremental learning. Experiments conducted on the CICIoT2023 dataset demonstrate that the proposed approach successfully balances performance and adaptability. It maintains accurate detection capabilities across various classification tasks while remaining responsive to evolving attack patterns.

Data-driven forecasting methods have the problems of complex calculations, poor portability and need a large amount of training data, which limits the application of data-driven methods in small cities. This paper propos-es a traffic flow forecasting method using a Nonlinear AutoRegressive model with eXogenous variables (NARX model), which uses a dynamic neural network Focused Time-Delay Neural Network (FTDNN) with a Tapped Delay Line (TDL) structure as a nonlinear function. The TDL structure enables the FTDNN to have short-term memory capabilities. At the same time, before the data is input into the FTDNN, the use of trend decomposition or differential calculation on the traffic data sequence can make the NARX model maintain long-term predictive ca-pabilities. Compared with common nonlinear models, the FTDNN has structural advantages. It uses a simple TDL structure without the memory mechanism and the gated structure, which can reduce the parameters of the model and reduce the scale of data. Through the four-day data of Guilin City, the traffic volume forecast for five minutes is verified, and the performance of the NARX model is better than that of the SARIMA model and the Holt-Win-ters model.

Prediction of the dynamics of a backward-facing step flow using focused time-delay neural networks and particle image velocimetry data-sets

Semi-supervised learning is especially important in data mining applications because it can make use of plentiful unlabeled data to train the high-quality learning models. Semi-Supervised Support Vector Machine (S3VM) is a powerful semi-supervised learning model. However, the high computational cost and non-convexity severely impede the S3VM method in large-scale applications. Although several learning algorithms were proposed for S3VM, scaling up S3VM is still an open problem. To address this challenging problem, in this paper, we propose a new incremental learning algorithm to scale up S3VM (IL-S3VM) based on the path following technique in the framework of Difference of Convex (DC) programming. The traditional DC programming based algorithms need multiple outer loops and are not suitable for incremental learning, and traditional path following algorithms are limited to convex problems. Our new IL-S3VM algorithm based on the path-following technique can directly update the solution of S3VM to converge to a local minimum within one outer loop so that the efficient incremental learning can be achieved. More importantly, we provide the finite convergence analysis for our new algorithm. To the best of our knowledge, our new IL-S3VM algorithm is the first efficient path following algorithm for a non-convex problem (i.e., S3VM) with local minimum convergence guarantee. Experimental results on a variety of benchmark datasets not only confirm the finite convergence of IL-S3VM, but also show a huge reduction of computational time compared with existing batch and incremental learning algorithms, while retaining the similar generalization performance.

Traditional machine learning algorithms are focused on batch learning from a static data set or from a well-known distribution. However, these algorithms take a considerable amount of time to learn a large amount of training data and besides many of them are not able to deal with nonstationary distributions. Recent machine learning challenges require the capability of online learning in nonstationary environments. Thus, in this work we propose a new learning method, for single-layer neural networks, that introduces a forgetting function in an incremental learning algorithm. The algorithm employs a recursive formula in order to obtain the solution of a weighted least squares problem. The performance of the method is experimentally checked over different data sets. The proposed algorithm has demonstrated high adaptation to changes while maintaining a low consumption of computational resources.KeywordsNeural NetworksIncremental LearningNonstationary Learning

The development of improved satellite technology generates a huge amount of remote sensing data, these data play the crucial role in natural resource management. The land use and land cover (LULC) change intensely affects local environment, as well as the global environment. Therefore, the quantifiable knowledge about LULC changes occur in global scale is important to make effective planning for conservation and precise use of natural resources, that has motivated the scientists to develop the various land cover change detection techniques. In this paper, we have proposed neural network-based approach, i.e., focused time delay neural network (FTDNN)-based approach for land cover change detection, which is a time series prediction-based approach and detect the sudden change in the enhanced vegetation index (EVI) time series. The performance of the proposed method has been addressed by using quantitative and qualitative analysis techniques. For the quantitative evaluation, the proposed algorithm is applied to the standard synthetic data set, which are analogous to EVI time series data set. The performance result of the proposed method compares with the four previously existing data mining-based benchmark techniques. The analysis was shown that the FTDNN-based method significantly outperforms than other techniques. For qualitative analysis, the San Francisco Bay Area data set has been used, which comprises real EVI time series. The proposed FTDNN-based method is applied to the San Francisco Bay Area data set and observe the interesting land cover changes. These outcomes indicate the effectiveness of data mining techniques for the land cover change detection problem.

Complexity Analysis of Multilayer Perceptron Neural Network Embedded into a Wireless Sensor Network

With the development of deep convolutional neural networks in recent years, the network structure has become more and more complicated and varied, and there are very good results in pattern recognition, image classification, scene classification, and target tracking. This end-to-end learning model relies on the initial large dataset. However, many data are gradually obtained in practical situations, which contradict the deep learning of one-time batch learning. There is an urgent need for an incremental learning approach that can continuously learn new knowledge from new data while retaining what has already been learned. This paper proposes an incremental learning algorithm based on convolutional neural network and support vector data description. CNN and AM-Softmax loss function are used to represent and continuously learn image features. Support vector data description is used to construct multiple hyperspheres for new and old classes of images. Class-incremental learning is achieved by the increment of hyperspheres. The experimental results show that the incremental learning method proposed in this paper can effectively extract the latent features of the image and adapt it to the learning situation of the class-increment. The recognition accuracy is close to batch learning.

Breast cancer is the most common kind of cancer, which is the cause of death among the women worldwide. There is evidence that shows that the early detection and treatment can increase the survival rate of patients who suffered this disease. Therefore, this paper proposes an automatic breast cancer diagnosis technique using a genetic algorithm for simultaneous feature selection and parameter optimization of an Multi Layer Perceptron (MLP) neural network. The aim of this paper is to propose a hybrid classification algorithm based on Multi-stage Weights Adjustment in the MLP (MWAMLP) neural network in two parts to improve the breast cancer diagnosis. In the first part, the three classifiers are trained simultaneously on the learning dataset. The output of the first part classifier together with the learning dataset is placed in a new dataset. This dataset uses a hybrid classifier method to model the mapping between the outputs of each ordinary classifier of the first part with real output labels. The proposed algorithm is implemented with three different variations of the backpropagation (BP) technique, namely the Levenberg–Marquardt, resilient BP and gradient descent with momentum for fine tuning of the weight of MLP neural network and their performances are compared. Interestingly, one of the proposed algorithms titled MWAMLP-RP produces the best and on average, 99.35% and 98.74% correct classification, respectively, on the Wisconsin Breast Cancer Database dataset, which is comparable with the obtained results from the methods titled GP-DLNN, GAANN and CAFS and other works found in the literature.

We applied multilayer perceptron (MLP) and radial basis function (RBF) neural networks using data from two water quality monitoring stations at the Karaj Dam in Iran. Input data were calcium ions (Ca2+), magnesium ions (Mg2+), sodium ions (Na+), chloride ions (Cl−), sulfate ( SO 42−), and pH, and the output data were total dissolved solids (TDS). An MLP with one hidden layer containing eight neurons was selected for the upstream water quality station using normalized input data. We developed a second MLP neural network for the downstream station with one hidden layer containing 10 neurons in the hidden layer using normalized input data. Considering applying normalized input data and one hidden layer, the coefficient of determination (R2) and index of agreement (IA) between the observed and the predicted data for the upstream and downstream monitoring stations using the MLP neural networks were 0.985, 0.84, 0.99, and 0.92, respectively. The RBF neural network with 100 neurons in its hidden layer reached the minimum errors between the observed and the predicted results in upstream and downstream stations. The R2 between observed and predicted data for upstream and downstream monitoring stations for the RBF was 0.999 and 0.998, respectively. Data normalization improved the performance of the MLP neural networks. Sensitivity analysis indicated that magnesium is the most effective water quality parameter for predicting TDS, and sulfate is the second most effective water quality parameter affecting TDS prediction at the Karaj Dam.

We applied multilayer perceptron (MLP) and radial basis function (RBF) neural network in upstream and downstream water quality stations of the Karaj Reservoir in Iran. For both neural networks, inputs were pH, turbidity, temperature, chlorophyll-a, biochemical oxygen demand (BOD) and nitrate, and the output was dissolved oxygen (DO). We used an MLP neural network with two hidden layers, for upstream station 15 and 33 neurons in the first and second layers respectively, and for the downstream station, 16 and 21 neurons in the first and second hidden layer were used which had minimum amount of errors. For learning process 6-fold cross validation were applied to avoid over fitting. The best results acquired from RBF model, in which the mean bias error (MBE) and root mean squared error (RMSE) were 0.063 and 0.10 for the upstream station. The MBE and RSME were 0.0126 and 0.099 for the downstream station. The coefficient of determination (<TEX>$R^2$</TEX>) between the observed data and the predicted data for upstream and downstream stations in the MLP was 0.801 and 0.904, respectively, and in the RBF network were 0.962 and 0.97, respectively. The MLP neural network had acceptable results; however, the results of RBF network were more accurate. A sensitivity analysis for the MLP neural network indicated that temperature was the first parameter, pH the second and nitrate was the last factor affecting the prediction of DO concentrations. The results proved the workability and accuracy of the RBF model in the prediction of the DO.

Research into the problem of classification of radar returns from the ionosphere has been taken up as a challenging task for the neural networks (NNs). It appears from the literature review that for the Multi layer Perceptron (MLP) NN trained with backpropagation, reported average classification accuracy was about 96% on the test instances. This paper investigates and designs an optimal classifier using a radial basis function (RBF) NN. Authors compare the performance of two NN configurations, namely a well-known MLP NN model and the proposed RBF NN model on the radar dataset collected from the published studies. It is shown that the proposed RBF NN, consistently, has 100% accuracy on bad instances and 99.1935% accuracy on good instances. The results show that the proposed RBF NN classifier clearly outperforms the MLP NN one in various performance measures such as MSE, NMSE, correlation coefficients, area under the ROC curve and classification accuracy on the testing datasets even after attempting different data partitions.

The prediction accuracy and generalization ability of neural/neurofuzzy models for chaotic time series prediction highly depends on employed network model as well as learning algorithm. In this study, several neural and neurofuzzy models with different learning algorithms are examined for prediction of several benchmark chaotic systems and time series. The prediction performance of locally linear neurofuzzy models with recently developed Locally Linear Model Tree (LoLiMoT) learning algorithm is compared with that of Radial Basis Function (RBF) neural network with Orthogonal Least Squares (OLS) learning algorithm, MultiLayer Perceptron neural network with error back-propagation learning algorithm, and Adaptive Network based Fuzzy Inference System. Particularly, cross validation techniques based on the evaluation of error indices on multiple validation sets is utilized to optimize the number of neurons and to prevent over fitting in the incremental learning algorithms. To make a fair comparison between neural and neurofuzzy models, they are compared at their best structure based on their prediction accuracy, generalization, and computational complexity. The experiments are basically designed to analyze the generalization capability and accuracy of the learning techniques when dealing with limited number of training samples from deterministic chaotic time series, but the effect of noise on the performance of the techniques is also considered. Various chaotic systems and time series including Lorenz system, Mackey-Glass chaotic equation, Henon map, AE geomagnetic activity index, and sunspot numbers are examined as case studies. The obtained results indicate the superior performance of incremental learning algorithms and their respective networks, such as, OLS for RBF network and LoLiMoT for locally linear neurofuzzy model.

Deep neural networks can be used to perform nonlinear operations at multiple levels, such as a neural network that is composed of many hidden layers. Although deep learning approaches show good results, they have a drawback called catastrophic forgetting, which is a reduction in performance when a new class is added. Incremental learning is a learning method where existing knowledge should be retained even when new data is acquired. It involves learning with multiple batches of training data and the newer learning sessions do not require the data used in the previous iterations. The Bayesian approach to incremental learning uses the concept of the probability distribution of weights. The key idea of Bayes theorem is to find an updated distribution of weights and biases. In the Bayesian framework, the beliefs can be updated iteratively as the new data comes in. Bayesian framework allows to update the beliefs iteratively in real-time as data comes in. The Bayesian model for incremental learning showed an accuracy of 82%. The execution time for the Bayesian model was lesser on GPU (670 s) when compared to CPU (1165 s).

Learning in neural networks can broadly be divided into two categories, viz., off-line (or batch) learning and online (or incremental) learning. In this paper, a review of a variety of supervised neural networks with online learning capabilities is presented. Specifically, we focus on articles published in main indexed journals in the past 10 years (2003---2013). We examine a number of key neural network architectures, which include feedforward neural networks, recurrent neural networks, fuzzy neural networks, and other related networks. How the online learning methodologies are incorporated into these networks is exemplified, and how they are applied to solving problems in different domains is highlighted. A summary of the review that covers different network architectures and their applications is presented.