Allocation of simulation effort for neural network vs. regression metamodels

Metamodels are models of simulation models. Metamodels are able to estimate the simulation responses corresponding to a given combination of input variables. A simulation metamodel is easier to manage and provides more insights than simulation alone. Traditionally, the multiple regression analysis is utilized to develop the metamodel from a set of simulation experiments. Simulation can consequentially benefit from the metamodelling in post-simulation analysis. A backpropagation (BP) neural network is a proven tool in providing excellent response predictions in many application areas and it outperforms regression analysis for a wide array of applications. In this paper, a BP neural network is used to generate metamodels for simulated manufacturing systems. For the purpose of optimal manufacturing systems design, mathematical models can be formulated by using the mapping functions generated from the neural network metamodels. The optimization model is then solved by a stochastic local search approach, simulated annealing (SA), to obtain an optimal configuration with respect to the objective of the systems design. Instead of triggering the detailed simulation programs, the SA-based optimization procedure evaluates the simulation outputs by the neural network metamodels. By using the SA-based optimization algorithm, the solution space of the studied problem is extensively exploited to escape the entrapment of local optima while the number of time consuming simulation runs is reduced. The proposed methodology is illustrated to be both effective and efficient in solving a manufacturing systems design problem through an example.

Metamodeling of large-scale simulations consisting of a large number of input parameters can be very challenging. Neural Networks have shown great promise in fitting these large-scale simulations even without performing factor screening. However, factor screening is an effective method for logically reducing the dimensionality of an input space and thus enabling more feasible metamodel calibration. Applying factor screening methods before calibrating Neural Network metamodels or any metamodel can have both positive and negative effects. The critical assumption for factor screening under investigation involves the prevalence of two-way interactions that contain a variable without a significant main effect by itself. In a simulation with a large parameter space, the prevalence of two-way interactions and their contribution to the total variability in the model output is far from transparent. Important questions therefore arise regarding factor screening and Neural Network metamodels: (a) is this a process worth doing with today's more powerful computing processors, which provide a larger library of runs to do metamodeling; and (b), does erroneously screening these buried interaction terms critically impact the level of metamodel fidelity that one can achieve. In this paper we examine these questions through the construction of a case study on a large-scale simulation. This study projects regional homelessness levels per county of interest based on a large array of budget decisions and resource allocations that expand out to hundreds of input parameters.

This study aims at developing abstract metamodels for approximating highly nonlinear relationships within a metal casting plant. Metal casting product quality nonlinearly depends on many controllable and uncontrollable factors. For improving the productivity of the system, it is vital for operation planners to predict in advance the amount of high quality products. Neural networks metamodels are developed and applied in this study for predicting the amount of saleable products. Training of metamodels is done using the Levenberg-Marquardt and Bayesian learning methods. Statistical measures are calculated for the developed metamodels over a grid of neural network structures. Demonstrated results indicate that Bayesian-based neural network metamodels outperform the Levenberg-Marquardt-based metamodels in terms of both prediction accuracy and robustness to the metamodel complexity. In contrast, the latter metamodels are computationally less expensive and generate the results more quickly.

Neural network metamodeling for cycle time-throughput profiles in manufacturing

A point cloud deep neural network metamodel method for aerodynamic prediction

A study is performed on the effectiveness of assessing accuracies of radial basis functions (RBFs) at design (or sampling) points. The RBF has been shown to be appropriate for creating approximation functions for linear, quadratic, and higher‐order nonlinear responses even with a relatively small size of design samples. One favourable feature of RBFs is that all RBF models pass exactly through design points; therefore, an RBF model has no error at design points. However, this feature prevents the use of statistical parameters, which are commonly used in polynomial‐regression approximations, to obtain model accuracy based on evaluations at design points. Since obtaining additional samples for many complex engineering applications is expensive, it is desirable to have some knowledge of the model accuracy based on evaluations at design points so that the best RBF model(s) can be selected. In this study, the effectiveness of using statistical parameter R2 for prediction (R) is evaluated for two highly accurate RBFs using mathematical functions and real engineering applications. The results show that R is inappropriate for RBF models, and should be used cautiously for highly nonlinear responses with a small number of sampling points and/or a large number of design variables. Copyright © 2006 John Wiley & Sons, Ltd.

Computer-based simulation models have been widely used to monitor the performance of complex systems. In spite of recent improvements in computers, use of simulation models may still be computationally impractical. On the other hand, metamodels can be used instead as an alternative approach, response surface metamodels (RSMs), particularly, polynomial regression metamodels (PRMs) being the most commonly preferred ones. However, existence of extreme observations in data may cause statistically invalid inferences if some remedial actions are not taken. One of the effective remedies is to use robust regression metamodels (RRMs), which decrease sensitivity of the results to such data. In this study, performances of several RRMs are compared with those of RSMs with respect to several criteria including accuracy, stability, efficiency and robustness using a validation method for the ballistic performance functions of a solid rocket engine. Furthermore, the effect of sampling strategies on the performance of these metamodels is assessed by collecting data using some traditional and computational design of experiment (DOEs). Results indicate that the performances of RRMs are competing with that of RSM, especially, in computational DOEs. Besides, they are more efficient and do not require an expert support with a capable software.

In this paper we present a new multiscale approach to history matching assisted by a neural network metamodel. The method starts with the construction of a fine scale a priori model that includes geological and geostatistical information. We then apply a singular value decomposition (SVD) in order to obtain a parametric representation of the permeability field, in a way that a fixed set of eigenimages are determined with the parameters to be inverted as weights in the expansion. Through this procedure not only is the number of parameters significantly reduced, but also the weights in the SVD expansion define a hierarchy that naturally separates the different resolution scales in the system. We show that the multiscale procedure alone helps to significantly reduce the CPU time required to accomplish the parameter estimation. Furthermore, the reduced parameter space facilitated the training of the neural network engine.

Polymer gel technologies have been widely employed in conformance-control applications to improve sweep and recovery from high permeability reservoirs. Modeling of polymer gel propagation, gelation time, and adsorption in the porous media requires a complex full-order computation of diffusivity equations, heat transfer and chemical reactions for components partitioning in oleic and aqueous phases. This paper presents a soft-computing alternative and data-driven approach to using machine learning to model deep polymer gel treatments in fractured reservoirs. Reference simulation models of various dual-permeability systems were used to generate the dataset for the machine learning model, an artificial neural network. A second-order reaction scheme for gel formulation was used to describe the chemical reaction between the polymer, polyacrylamide, and the cross-linker component, chromium acetate. Viscosities, adsorption properties and residual resistance factors of the polymer and produced gel were populated based on experimental data. Various conformance design factors and reservoir properties were parameterized for inclusion in the neural network, including: temperature, injection rate, gel concentration, bottom-hole pressure, drainage radius, porosity, fracture spacing and permeability. Feature analysis of the input variables indicated that ten parameters are sufficient to train the model and predict the performance of the conformance treatment with indicators including the oil and water rate profiles improvement after applying the polymer gel treatment. The dataset was randomly divided into 80% training, 10% validation and 10% testing sets. Early stopping and monitoring of the validation and testing set's errors were used to generalize the solution and enhance the performance of the neural network. Hyperparameter tuning showed that using a deep multi-hidden layers neural network was more effective than increasing the neurons in a single hidden layer. The weights and biases of the model were adjusted using a mean squared error loss function and a gradient descent optimizer. A correlation of 90% was achieved for the test samples with a mean absolute deviation of less than 10% for all modeled variables. The developed neural network model was able to reduce the complexity of the full-order simulation model by accurately predicting the performance of polymer gel treatments in fractured reservoirs at 200 times faster speed than commercial simulators with only ten input variables of polymer gel design and reservoir properties. This work presents a unique surrogate modeling approach based on machine learning to describe complex polymer gel kinetics and flow dynamics in deep conformance applications. The presented neural network model can be used to robustly predict the oil recovery performance of polymer gel treatments in fractured reservoirs outperforming commercial simulators in terms of computational complexity and processing speed.

Random search algorithms are often used to solve discrete optimization-via-simulation (DOvS) problems. The most critical component of a random search algorithm is the sampling distribution that is used to guide the allocation of the search effort. A good sampling distribution can balance the trade-off between the effort used in searching around the current best solution (which is called exploitation) and the effort used in searching largely unknown regions (which is called exploration). However, most of the random search algorithms for DOvS problems have difficulties in balancing this trade-off in a seamless way. In this paper we propose a new scheme that derives a sampling distribution from a fast fitted Gaussian process based on previously evaluated solutions. We show that the sampling distribution has the desired properties and can automatically balance the exploitation and exploration trade-off. Furthermore, we integrate this sampling distribution into a random research algorithm, called a Gaussian process-based search (GPS) and show that the GPS algorithm has the desired global convergence as the simulation effort goes to infinity. We illustrate the properties of the algorithm through a number of numerical experiments.

In the analysis of a manufacturing system, the analyst is often interested in the change in mean cycle time as a function of different throughput (start rate) levels. Since the variance of the mean cycle time generally increases as the start rate increases, an equal allocation of simulation effort at each simulated throughput level will result in confidence intervals of different widths. This paper discusses an approach for generating nearly equal sized mean cycle time confidence intervals at selected throughput levels when the computing budget is fixed (limited). The simulation effort allocation procedure described in this paper determines the proportion of the fixed budget to allocate at each throughput level based on the asymptotic variance.

Background. The disadvantage of spectrally efficient signals with frequency multiplexing is the occurrence of intersymbol interference, which is further aggravated when these signals propagate in frequency selective channels.
 Aim. The possibility and effectiveness of using neural network approaches for channel equalization and signal detection in communication systems using SEFDM signals has been assessed.
 Methods. A receiver structure for SEFDM systems based on a deep complex-valued convolutional neural network is proposed, which allows recovering bits from the temporal representation of the signal without using the fractional Fourier transform and inverting the cross-correlation matrix between frequency subcarriers. A two-stage network training scheme has been developed. Based on simulation modeling, a comparative analysis of the noise immunity of SEFDM systems was carried out both in a channel with white Gaussian noise and in a channel with Rayleigh fading, using classical and neural network receivers.
 Results. It is shown that there is no loss of noise immunity in channels with additive white Gaussian noise and an increase in noise immunity of the system up to 2 dB in the channel specified by the extended automotive model (3GPP-EVA).
 Conclusion. The effectiveness of using deep neural complex-valued convolutional networks as receivers for spectrally efficient communication systems, as well as their advantage over classical ones, is shown.
 Keywords – SEFDM; deep complex-valued convolutional neural network; turbocoding.

Nonlinear identification via connected neural networks for unsteady aerodynamic analysis

The research technique of human motor nerve is more complicated. In order to improve the understanding level of human motor nerve structure, a fusion architecture of human motion nerve and neural network computer driven by education are proposed. The human motor nerve simulation model is established by using the computer data simulation model. The model proposes an optimization scheme from the algorithm flow, the data transformation technology used to improve the real-time performance of the neural network and enhance the dynamic capture of the motor nerve changes. In order to further improve the architecture of the neural network and computer architecture driven by sports, a neural network algorithm is added to realize the sequence optimization of data to test the authenticity and efficiency of the human neural network simulation model neural network algorithm. In this paper, the feasibility and functionality of the algorithm are tested with comparative experiments. The results show that the neural network algorithm neural network algorithm not only has a calculation time of only 12 seconds. Moreover, the calculation accuracy is high, achieving a high level of accuracy of 98%. On the other hand, other algorithms have lower accuracy and longer calculation time. The research shows that the neural network algorithm can improve the human motor nerve capture and optimize the architecture, which can provide reference for the fusion of human motion nerve and computer technology in the future.

As the gas path components’ performance cannot avoid degeneration while the aero-engine is working, the related fault diagnosis technology is of great meaning to the aero-engine’s health management system. Aiming at the aero-engine’s nonlinear model at the design point, this paper optimizes the selection rules of the neural network’s input parameters and output parameters through relative analysis of PCC (Pearson correlation coefficient). For the regular BP (back propagation) neural network’s weakness of an unstable convergence speed and easy to fall into minimum value, this paper designs a new method via self-adaptive neural network that can accurately estimate the degeneration level of the aero-engine. This method combines scale factor and momentum factor, improves the learning speed of network and enhances the confidence level of neural network as well as the generalization ability of engine’s model parameters. The results prove that the self-adaptive neural network method proposed in this paper has better accuracy than the regular BP neural network, and when the training samples are not too many, the training can also produce an ideal network, which guarantees a good accuracy for the fault diagnosis of the aero-engine’s health parameters.