Machine Learning Regression Techniques for the Modeling of Complex Systems: An Overview

This article deals with the development of a probabilistic surrogate model for the uncertainty quantification of the voltage output spectral envelope of a power converter with several stochastic parameters. The proposed approach relies on the combination of the least-squares support vector machine (LS-SVM) regression with the Gaussian process regression (GPR), but it can suitably be applied to any deterministic regression techniques. As a first step, the LS-SVM regression is used to build an accurate and fast-to-evaluate deterministic model of the system responses starting from a limited set of training samples provided by the full-computational model. Then the GPR is used to provide a probabilistic model of the regression error. The resulting LS-SVM+GPR probabilistic model not only approximates the system responses for any configuration of its input parameters, but also provides an estimation of its prediction uncertainty, such as the confidence intervals (CIs). The above technique has been applied to qualify the uncertainty of the spectral envelope of the output voltage of a buck converter with 17 independent Gaussian parameters. The feasibility and the accuracy of the resulting model have been investigated by comparing its predictions and CI with the ones obtained by five different surrogate models based on state-of-the-art techniques and by the reference Monte Carlo results.

Particulate matter (PM) emissions, including soot, pose a significant environmental concern. Traditional emission control approaches, such as engine calibration, often struggle to maintain consistent performance across varying driving conditions, engine speeds, and loads. This inconsistency leads to variations in pollutant output, making it challenging for internal combustion engines to comply with stringent emission standards under real-world driving conditions. Physics-based soot models, grounded in complex mathematical equations, have been developed to capture the mechanisms of soot formation. However, these models often face accuracy limitations and require substantial computational resources, rendering them impractical for real-time emission control. In contrast, Machine Learning (ML) techniques offer a more versatile and efficient approach to soot prediction. This paper introduces a machine learning methodology for predicting Soot Particle Numbers (PN). The approach integrates both physics-based and black-box strategies to create a grey-box model for predicting soot emissions from a 1600 cc Gasoline Direct Injection (GDI) engine across diverse load and speed conditions. The physical model is based on a validated 1D engine model in GT-SUITE, while the black-box model is formulated using multiple ML approaches exclusively on measured data. Ultimately, the grey-box model is developed through ML techniques based on measured data and combustion-related parameters obtained from the 1D engine physical model in GT-SUITE. The grey-box model’s results are compared with those of the black-box model. The ML techniques employed in this study include Gaussian Process Regression (GPR), Support Vector Machine (SVM), Artificial Neural Network (ANN), and an Ensemble of Regression Trees (ERT). The Minimum Redundancy Maximum Relevance (MRMR) and RreliefF feature selection methods are used to identify the most crucial features in soot PN prediction. The results indicate that the grey-box GPR exhibits superior performance for both single and multi-pulse injection strategies, achieving a coefficient of determination (R2) of 0.99.

Machine Learning (ML) techniques are pervasive tools in various emerging commercial applications, but have to be accommodated by powerful computer systems to process very large data. Although general-purpose CPUs and GPUs have provided straightforward solutions, their energy-efficiencies are limited due to their excessive supports for flexibility. Hardware accelerators may achieve better energy-efficiencies, but each accelerator often accommodates only a single ML technique (family). According to the famous No-Free-Lunch theorem in the ML domain, however, an ML technique performs well on a dataset may perform poorly on another dataset, which implies that such accelerator may sometimes lead to poor learning accuracy. Even if regardless of the learning accuracy, such accelerator can still become inapplicable simply because the concrete ML task is altered, or the user chooses another ML technique.In this study, we present an ML accelerator called PuDianNao, which accommodates seven representative ML techniques, including k-means, k-nearest neighbors, naive bayes, support vector machine, linear regression, classification tree, and deep neural network. Benefited from our thorough analysis on computational primitives and locality properties of different ML techniques, PuDianNao can perform up to 1056 GOP/s (e.g., additions and multiplications) in an area of 3.51 mm^2, and consumes 596 mW only. Compared with the NVIDIA K20M GPU (28nm process), PuDianNao (65nm process) is 1.20x faster, and can reduce the energy by 128.41x.

This paper addresses processing of spectrophotometric array signals based on genetic algorithms (GA) least square support vector machines (LS-SVM) regression to provide a powerful model for machine learning and data mining. The key to complete LS-SVM regression is to choose its optimal parameters. Due to their outstanding ability in solving global optimization problems in complex multidimensional search space, GA are used in this study to obtain the optimal parameter combination of the LS-SVM model. Experimental results showed the GA-LS-SVM method to be successful for simultaneous multicomponent determination even where severe overlap of spectra was present. KeywordsLeast squares support vector machines; Genetic algorithms; Spectrophotometric array signals; Overlapping spectra; Artificial intelligence Nowadays, with the application of photometric diode array detector and computers, rapid scanning commercial spectrophotometers are capable of quickly generating huge data consisting of hundreds and even thousands of absorbance values per spectrum. The array data named fullspectrum contain sufficient information to be able to determine the contents of various compounds. The main drawback of ultraviolet-visible (UV-VIS) is its poor selectivity because in many cases UV-VIS spectra display strong overlaps, especially some less specific and selective chromagenic reagents often give rise to strongly overlapped spectra in many cases. The combination of artificial intelligence methods with the computer-controlled spectrophotometers was proven to be effective in overcoming this difficulty [1-3]. Artificial neural network (ANN) is a form of artificial intelligence that mathematically simulates biological nervous system [4, 5]. However, ANN often has slow convergence, is prone to the existence of many local minima during training, and has a tendency of overfitting. Recently, a promising technology called support vector machines (SVM) has been used for classification and regression problems. SVM pioneered by Vapnik is a kind of machine learning method based on modern statistical learning theory and has notable properties including absence of local minima and high generalization ability [6, 7]. Suykens and his coworkers [8] introduced a modified version of SVM called least square SVM (LS-SVM) , which requires solving a set of linear equations instead of a quadratic programming problem and is much easier and computationally simpler than SVM. SVM and LS-SVM represent relatively recent artificial intelligence method and have found some applications in image analysis, classification and disease diagnosis etc. [9, 10]. It is worth mentioning that the success of LS-SVM model is highly dependent on the optimum choice of two parameters, the relative weight of regression error γ and the kernel width σ of radial basis function (RBF). Genetic algorithms (GA) [11, 12] introduced by John Holland are probabilistic optimization techniques based on natural evolution and genetics and Darwin’s theory of survival of the best. With their efficient and robust global search ability, GA are used to search two optimal parameters for the LS-SVM model simultaneously and automatically. The LS-SVM model then performs the regression task using these optimal parameters.

Landslide disaster risk reduction necessitates the investigation of different geotechnical causal factors for slope failures. Machine learning (ML) techniques have been proposed to study causal factors across many application areas. However, the development of ensemble ML techniques for identifying the geotechnical causal factors for slope failures and their subsequent prediction has lacked in literature. The primary goal of this research is to develop and evaluate novel feature selection methods for identifying causal factors for slope failures and assess the potential of ensemble and individual ML techniques for slope failure prediction. Twenty-one geotechnical causal factors were obtained from 60 sites (both landslide and non-landslide) spread across a landslide-prone area in Mandi, India. Relevant causal factors were evaluated by developing a novel ensemble feature selection method that involved an average of different individual feature selection methods like correlation, information-gain, gain-ratio, OneR, and F-ratio. Furthermore, different ensemble ML techniques (Random Forest (RF), AdaBoost (AB), Bagging, Stacking, and Voting) and individual ML techniques (Bayesian network (BN), decision tree (DT), multilayer perceptron (MLP), and support vector machine (SVM)) were calibrated to 70% of the locations and tested on 30% of the sites. The ensemble feature selection method yielded six major contributing parameters to slope failures: relative compaction, porosity, saturated permeability, slope angle, angle of the internal friction, and in-situ moisture content. Furthermore, the ensemble RF and AB techniques performed the best compared to other ensemble and individual ML techniques on test data. The present study discusses the implications of different causal factors for slope failure prediction.

ABSTRACTThis paper reviews the activity-travel behaviour literature that employs Machine Learning (ML) techniques for empirical analysis and modelling. Machine Learning algorithms, which attempt to build intelligence utilizing the availability of large amounts of data, have emerged as powerful tools in the fields of pattern recognition and big data analysis. These techniques have been applied in activity-travel behaviour studies since the early ’90s when Artificial Neural Networks (ANN) were employed to model mode choice decisions. AMOS, an activity-based modelling system developed in the mid-’90s, has ANN at its core to model and predict individual responses to travel demand management measures. In the dawn of 2000, ALBATROSS, a comprehensive activity-based travel demand modelling system, was proposed by Arentze and Timmermans using Decision Trees. Since then researchers have been exploring ML techniques like Support Vector Machines (SVM), Decision Trees (DT), Neural Networks (NN), Bayes Classifiers, and more recently, Ensemble Learners to model and predict activity-travel behaviour. A large number of publications over the years and an upward trend in the number of published articles over time indicate that Machine Learning is a promising tool for activity-travel behaviour analysis and prediction. This article, first of its kind in the literature, reviews these studies and explores the trends in activity-travel behaviour research that apply ML techniques. The review finds that mode choice decisions have received wide attention in the literature on ML applications. It was observed that most of the studies identify the lack of interpretability as a serious shortcoming in ML techniques. However, very few studies have attempted to improve the interpretability of the models. Further, some studies report the importance of feature engineering in ML-based studies, but very few studies adopt feature engineering before model development. Spatiotemporal transferability of models is another issue that has received minimal attention in the literature. In the end, the paper discusses possible directions for future research in the area of activity-travel behaviour modelling using ML techniques.

There are many parameters, which affect the dynamic response of structures, such as the type of structure, type of foundation, soil characteristics, etc. In this paper, soil-structure interaction in the analysis and design of reinforced concrete framed buildings has been investigated using SAP2000 vs16. Base shear, bending moment, deflection and natural period of multi-storey buildings with raft foundations, having different height to width ratios, resting on different types of soils, and subjected to a seismic ground motion were studied using Gaussian process regression (GPR) and least-square support vector machine (LSSVM). The ground motions were selected from the data of the Bhuj earthquake of 2001. A comparison of the performance of the GPR model and LSSVM model has been done based on Probability Distribution Function (PDF) and Cumulative Distribution Function (CDF).The results showed that the developed bending moment model was found more efficient compared to base shear and deflection models.

A novel method based on combination of highly sensitive spectrofluorimetry and flexible the machine learning techniques was proposed for the simultaneous spectrofluorimetric determination of α-naphthol, β-naphthylamine and carbazole with overlapping peaks. This method addresses multivariate calibration based on the least square support vector machines (LS-SVM) regression to provide a powerful model for machine learning and data mining. The LS-SVM technique has the advantages to offer the capability of learning a high dimensional feature with fewer training data, and to decrease the computational complexity by only requiring to solve a set of linear equations instead of a quadratic programming problem. Experimental results showed the LS-SVM method to be successful for simultaneous multicomponent determination even where severe overlap of spectra was present. The relative standard errors of prediction (RSEP) obtained for total components using LS-SVM and PLS were compared. It is found that the LS-SVM method is better than the conventional PLS methods.

Constant power loads (CPLs) in the DC microgrids will lead to the instability of the bus voltage, so the power variation range needs to be limited. In this paper, a based on machine learning critical value prediction method is proposed for CPLs. Firstly, Pearson correlation analysis is used to find the factors that have effects on CPLs critical value in terms of droop coefficient and bus voltage. Then, support vector machine and Gaussian process regression prediction model of CPLs critical value are established. Finally, different scenarios of DC microgrid are established to verify the proposed algorithms. The results show that machine learning algorithms can accurately predict the critical value of CPLs, and compared with support vector machine, Gaussian process regression method has higher prediction accuracy and universality.

Machine learning (ML) techniques learn a system by observing it. Events and occurrences in the network define what is expected of the network’s operation. It is for this reason that ML techniques are used in the computer network security field to detect unauthorized intervention. In the event of suspicious activity, the result of the ML analysis deviates from the definition of expected normal network activity and the suspicious activity becomes apparent. Support vector machines (SVM) are ML techniques that have been used to profile normal network activity and classify it as normal or abnormal. They are trained to configure an optimal hyperplane that classifies unknown input vectors’ values based on their positioning on the plane. We propose to use SVM models to detect malicious behavior within low-power, low-rate and short range networks, such as those used in the Internet of Things (IoT). We evaluated two SVM approaches, the C-SVM and the OC-SVM, where the former requires two classes of vector values (one for the normal and one for the abnormal activity) and the latter observes only normal behavior activity. Both approaches were used as part of an intrusion detection system (IDS) that monitors and detects abnormal activity within the smart node device. Actual network traffic with specific network-layer attacks implemented by us was used to create and evaluate the SVM detection models. It is shown that the C-SVM achieves up to 100% classification accuracy when evaluated with unknown data taken from the same network topology it was trained with and 81% accuracy when operating in an unknown topology. The OC-SVM that is created using benign activity achieves at most 58% accuracy.

Recent progress on machine learning techniques for flow field optimization toward performance and commercial scale proton exchange membrane fuel cell: A systematic review

In this study, six machine learning (ML) models, namely, random forest (RF), Gaussian process regression (GPR), support vector regression (SVR), decision tree (DT), least squares support vector machine (LSSVM), and multivariate adaptive regression spline (MARS) models, were employed to reconstruct the missing daily‐averaged discharge in a mega‐delta from 1980 to 2015 using upstream‐downstream multi‐station data. The performance and accuracy of each ML model were assessed and compared with the stage‐discharge rating curves (RCs) using four statistical indicators, Taylor diagrams, violin plots, scatter plots, time‐series plots, and heatmaps. Model input selection was performed using mutual information and correlation coefficient methods after three data pre‐processing steps: normalization, Fourier series fitting, and first‐order differencing. The results showed that the ML models are superior to their RC counterparts, and MARS and RF are the most reliable algorithms, although MARS achieves marginally better performance than RF. Compared to RC, MARS and RF reduced the root mean square error (RMSE) by 135% and 141% and the mean absolute error by 194% and 179%, respectively, using year‐round data. However, the performance of MARS and RF developed for the climbing (wet season) and recession (dry season) limbs separately worsened slightly compared to that developed using the year‐round data. Specifically, the RMSE of MARS and RF in the falling limb was 856 and 1,040 m3/s, respectively, while that obtained using the year‐round data was 768 and 789 m3/s, respectively. In this study, the DT model is not recommended, while the GPR and SVR models provide acceptable results.

Diabetes is a major chronic syndrome caused by a series of metabolic abnormalities in which blood glucose levels are abnormally high for an indeterminate amount of time. It influences various organs in the human body, resulting in a variety of complex diseases such as stroke, renal disease, pulmonary embolism, eyesight, and so on. Diabetes Disorders (DD) are presently one of the healthcare top causes of mortality. Predictive analytics in the health care system is a huge obstacle, but if accurate early prediction is achieved, the potential risk and degree of diabetes may be significantly decreased. Machine learning (ML) techniques are now used to analyze medical datasets at an earlier stage of life in keeping people safe. In this research, we utilized several ML approaches notably Logistic Regression, Decision Tree (DT), XGBoost, Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Random Forest (RF) on PIMA Indian Diabetes Dataset in order to monitor and evaluate their performances in diabetes prediction. The performance of the various ML algorithms employed in this research suggests which algorithm is most suitable in diabetes prediction. It is observed that among all the models XGBoost had outperformed the other ML techniques with an accuracy of 80.73% while SVM was the second-best performing model with a classification accuracy of 80.21%. Thus, employing ML techniques, this study aims to assist doctors as well as clinicians in the early detection of diabetes.

The software quality is identified by the use of a tool, which is defined as Software Reliability (SR). In many applications such as nuclear power plants, home applications, space missions, personal computers and, so on, software plays an important role in human lives over the few decades. Nowadays, the interactions of inter-component of modular software systems are very difficult to analyze by traditional SR methods due to the fast growth of complex software applications. Therefore, the SR prediction provides meticulous and remarkable results by developing the applications of Machine Learning (ML) techniques. In the research paper, the discussion of various ML techniques to predict the SR are presented along with the evaluation of selected performance criteria. According to the selected performance criteria, the evaluation of ML techniques to predict the SR based on the different datasets collected from industrial software, the prediction of SR are carried out by several ML techniques such as Support Vector Machines (SVM), Reduced Error Pruning Tree (REPTree), Instance-Based Learning, Neuro-fuzzy, General Regression Neural Network (GRNN), Linear Regression, Bagging, Feed Forward Back Propagation Neural Network (FFBPNN), and Multi-Layer Perceptions (MLP) are presented. To enhance the performance of techniques, this study helps the researchers effectively by discovering new meaningful knowledge about SR prediction.KeywordsBaggingMachine learningNeuro-fuzzySoftware reliabilitySoftware qualitySupport vector machine

For proper design and operation of biomass‐based energy systems, it is important to determine the higher heating value (HHV) of biomass. In this paper, two machine learning (ML) approaches, namely extra trees (ET) and least squares support vector machine (LSSVM), are used to predict the value of HHV associated with biofuels. The data required for HHV calculation, including proximate and ultimate analyses datasets, were collected from the literature. The performances of these two ML approaches for predicting biomass HHV were then compared with other smart models available in the literature. Even though the available empirical models can predict the biomass HHV with acceptable precision, it was found that our proposed ML techniques have a superior performance based on the error analysis; the proposed approaches also consider all key biomass characteristics in the developed models. In addition, the ET model proved to be slightly more accurate compared to the LSSVM model. Additionally, the developed proximate‐based ET model showed better performance compared to the ultimate‐based ET model. The most influential parameters in the developed ET models for the proximate and ultimate approaches were determined to be ash fraction and carbon fraction, respectively. Finally, it was concluded that the smart modelling techniques can be utilized as a robust and reliable alternative predictive methodology to replace direct laboratory measurement of the biomass HHV.