A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics

The current work outlines application of a data-driven multilayer perceptron-based artificial neural network (ANN) model to characterize the influence of melt compositions, tundish temperature, tundish superheat, casting speed and mould oscillation frequency on the important processing parameters such as mould powder consumption rate, oscillation mark depth and metallurgical length during continuous casting process. A two-layer feedforward back-propagation neural network model has been developed for predicting the probability of occurrence of defect in the cast product. The network training architecture has been optimized using a gradient-based algorithm, namely the back-propagation algorithm. The neural network predictions are found to be in good agreement with regard to oscillation mark depth, mould powder consumption rate, metallurgical length and probability of occurrence of defect using data obtained from an operating Indian steel plant (Rashtriya Ispat Nigam Limited, Visakhapatnam). The ANN model prediction has been validated successfully with multiple linear regression analysis carried out on each data set.

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher–student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.

Stability and Hopf bifurcation analysis of a simplified six-neuron tridiagonal two-layer neural network model with delays

An integrated approach based on genetic algorithm (GA) and an artificial neural network (ANN) is presented for structural optimization of a high power density magneto-gel brake. For this method, the GA method is employed for obtaining the optimal configuration of the brake by minimizing the dimensions of the brake. Subsequently, a two-layer BP neural network model is trained to obtain the correlation between main design parameters and performance of the brake, and then it is used to predict the performance of the magnetic gel brake with high power density. The coupled method incorporating the ANN with GA can reduce substantially the computation time during optimizing brake performance. Meanwhile, the proposed method’s validity is demonstrated by comparisons between the experiment and existing data.

1. In this study, neural network (NN) and response surface (RS) models were developed to investigate the response [average daily gain (ADG) and feed efficiency (FE)] of young broiler chickens to dietary protein and lysine. For this purpose, data on their responses to dietary protein and lysine were extracted from the literature and separate NN and RS models were constructed. 2. Comparison between the NN and RS models revealed higher accuracy of prediction with the NN models compared to the RS models. In terms of R 2 values, the NN models developed for both ADG (R 2 = 0.923) and FE (R 2 = 0.904) were far superior to the RS models (R 2 for ADG = 0.511; R 2 for FE = 0.67). This suggests that the NN models can serve as an alternative option to conventional regression approaches including use of RS models. 3. Optimisation of the NN models developed for response to protein and lysine showed that diets containing 220.7 (g/kg of diet) protein and 12.85 (g/kg of diet) lysine maximise ADG, whereas maximum FE is achieved with diets containing 241.3 and 13.12 (g/kg) protein and lysine, respectively. Based on the optimisation results, optimal dietary protein and lysine concentrations for maximum FE in broiler chickens during the starting period are higher than for ADG.

Modelling and Optimizing Pyrene Removal from the Soil by Phytoremediation using Response Surface Methodology, Artificial Neural Networks, and Genetic Algorithm

The molecular regulatory network (MRN) within a cell determines cellular states and transitions between them. Thus, modeling of MRNs is crucial, but this usually requires extensive analysis of time-series measurements, which is extremely difficult to obtain from biological experiments. However, single-cell measurement data such as single-cell RNA-sequencing databases have recently provided a new insight into resolving this problem by ordering thousands of cells in pseudo-time according to their differential gene expressions. Neural network modeling can be employed by using temporal data as learning data. In contrast, Boolean network modeling of MRNs has a growing interest, as it is a parameter-free logical modeling and thereby robust to noisy data while still capturing essential dynamics of biological networks. In this study, we propose a Boolean feedforward neural network (FFN) modeling by combining neural network and Boolean network modeling approach to reconstruct a practical and useful MRN model from large temporal data. Furthermore, analyzing the reconstructed MRN model can enable us to identify control targets for potential cellular state conversion. Here, we show the usefulness of Boolean FFN modeling by demonstrating its applicability through a toy model and biological networks.

Background: Nowadays, forecasting models based on nonparametric models have been developed in many branches of science such as mathematics and economics. However, the relatively complicated structure of these models has made them less practical in medical sciences. Objectives: In this study, we investigated the application of a nonparametric regression model to predict the psychological symptoms appearing six months after mild traumatic brain injury. We also made a comparison between the performance of nonparametric regression and artificial neural network models in predicting the psychological symptoms. Methods: In a six-month period during 2015 - 2017, information of 100 mild traumatic patients was included in a prospective cohort study. The data were then divided randomly into two groups of training (n = 50) and testing (n = 50) for 100 times. In the training group, the focus was on 100 artificial neural network and nonparametric regression models. However, in the testing group, a comparison was made between the values obtained using the two final models. To compare the models, the ROC curve and the accuracy rate were finally applied. Results: According to the obtained results, the nonparametric regression model showed an accuracy rate of 91.25% while the neural network model had an accuracy rate of 85.34%. In the experimental set for both neural network and nonparametric regression models, the area under the ROC curve appeared to be 81.51% and 85.73 %, respectively. Conclusions: The nonparametric regression model appeared to be more powerful than the neural network model in predicting psychological symptoms.

Background: Data mining (DM) is an approach used in extracting valuable information from environmental processes. This research depicts a DM approach used in extracting some information from influent and effluent wastewater characteristic data of a waste stabilization pond (WSP) in Birjand, a city in Eastern Iran. Methods: Multiple regression (MR) and neural network (NN) models were examined using influent characteristics (pH, Biochemical oxygen demand [BOD 5 ], temperature, chemical oxygen demand [COD], total suspended solids [TSS], total dissolved solid [TDS], electrical conductivity [EC] and turbidity) as the regression input vectors. Models were adjusted to input attributes, effluent BOD 5 (BODout) and COD (CODout). The models performances were estimated by 10-fold external cross-validation. An internal 5-fold cross-validation was also used for the training data set in NN model. The models were compared using regression error characteristic (REC) plot and other statistical measures such as relative absolute error (RAE). Sensitivity analysis was also applied to extract useful knowledge from NN model. Results: NN models (with RAE = 78.71 ± 1.16 for BODout and 83.67 ± 1.35 for CODout) and MR models (with RAE = 84.40% ± 1.07 for BODout and 88.07 ± 0.80 for CODout) indicate different performances and the former was better (P < 0.05) for the prediction of both effluent BOD 5 and COD parameters. For the prediction of CODout the NN model with hidden layer size (H) = 4 and decay factor = 0.75 ± 0.03 presented the best predictive results. For BODout the H and decay factor were found to be 4 and 0.73 ± 0.03, respectively. TDS was found as the most descriptive influent wastewater characteristics for the prediction of the WSP performance. The REC plots confirmed the NN model performance superiority for both BOD and COD effluent prediction. Conclusion: Modeling the performance of WSP systems using NN models along with sensitivity analysis can offer better understanding on exploring the most significant parameters for the prediction of system performance. The findings of this study could build the foundation for prospective work on the characterization of WSP operations and optimization of their performances with a view to conducting statistical approaches.

In this paper, a speech-to-text translation model has been developed for Malaysian speakers based on 41 classes of Phonemes. A simple data acquisition algorithm has been used to develop a MATLAB graphical user interface (GUI) for recording the isolated word speech signals from 35 non-native Malaysian speakers. The collected database consists of 86 words with 41 classes of phoneme based on Affricatives, Diphthongs, Fricatives, Liquid, Nasals, Semivowels and Glides, Stop and Vowels. The speech samples are preprocessed to eliminate the undesirable artifacts and the fuzzy voice classifier has been employed to classify the samples into voiced sequence and unvoiced sequence. The voiced sequences are divided into frame segments and for each frame, the Linear Predictive co-efficients features are obtained from the voiced sequence. Then the feature sets are formed by deriving the LPC features from all the extracted voiced sequences, and used for classification. The isolated words chosen based on the phonemes are associated with the extracted features to establish classification system input-output mapping. The data are then normalized and randomized to rearrange the values into definite range. The Multilayer Neural Network (MLNN) model has been developed with four combinations of input and hidden activation functions. The neural network models are trained with 60%, 70% and 80% of the total data samples. The neural network architecture was aimed at creating a robust model with 60%, 70%, and 80% of the feature set with 25 trials. The trained network model is validated by simulating the network with the remaining 40%, 30%, and 20% of the set. The reliability of trained network models were compared by measuring true-positive, false-negative, and network classification accuracy. The LPC features show better discrimination and the MLNN neural network models trained using the LPC spectral band features gives better recognition.

A comparative study of neural network and mechanistic models for surface complexation

Computer-aided Prediction of the ZrO2 Nanoparticles' Effects on Tensile Strength and Percentage of Water Absorption of Concrete Specimens

Changes on the CIELab values of the dyed materials after the different chemical finishing treatments using artificial neural network (ANN) and linear regression (LR) models have been predicted. The whole structural properties of fabrics and some process data which were from fiber to the finishing parameters were accepted as inputs in these models. The networks having different structures were established, and it was also focus on the parameters which could affect the performance of the established networks. It was determined that we could successfully predict the color differences values occurring on the material after the finishing applications. In addition, we realized that some ANN parameters affected the prediction performance while establishing the models. After training ANN models, the prediction of the color difference values was also tried by linear regression models. Then, extra ANN models were established for all outputs using the parameters as inputs in the LR equations, and the prediction performances of both established models were compared. According to the results, the neural network model gives a more accurate prediction performance than the LR models.

An attempt has been made to develop neural network and neuro-fuzzy based models to predict the green compressive strength of clay bonded moulding sand mixtures and to analyse the properties of the mixed sand. Data for the models were generated following standard experimental procedure in the sand laboratory. Process parameters such as clay content, moisture, coal dust and mulling time were varied. The collected data were used in developing these models. The predicted green compressive strengths by neural network and neuro-fuzzy based models have been found to be in good agreement with the experimental values. Furthermore, it has been observed from the results that the predicted values from the neuro-fuzzy model are more accurate than those predicted from the neural network model.

The use of neural networks promotes flexible non-linear modelling. Also, it is the perspective direction. The article investigates the influence of the main road characteristics on the accident rate. Twelve variables are selected as the main road characteristics. The result of modelling is the two-layer feed-forward neural network with hidden sigmoid nodes and linear output nodes. The operability of the model is estimated by calculating the index of regression by training, validation and testing. The results are acceptable for the intended purpose. The resulting model can be used for assessing and forecasting the safety rate on the two-lane roads with the intensity of 120 to 340 cars/hour. These include a significant part of the streets of CIS countries and other states. Besides, the neural network model can become the basis for the development of software products for assessing the accident rate based on road characteristics or can be implemented in existing software.