ANALYSIS OF TRAINING SET PARALLELISM FOR BACKPROPAGATION NEURAL NETWORKS

A multilayer feedforward neural network with a modification to the standard backpropagation training of neural nets is considered. Information on the gradients of target outputs with respect to network inputs is used in this modification. This modified neural network is then applied for approximating design requirements of aerospace composite components, such as an aircraft engine guide vane and a satellite reflector assembly. This neural network functional approximator is then used with an optimization algorithm for determining the optimal parameters for component design requirements. The modified neural network provides for faster convergence of learning process and requires a smaller number of hidden layer nodes for convergence to similar training error levels. HE application of neural networks as universal approximators has been the focus of tremendous amount of activity in the past few years.16 Neural networks have been used for approximating and modeling both memoryless systems and dynamical systems with memory. This paper does not go into the various neural net architectures and training schemes, and the interested reader could refer to standard texts on this subject matter.1'2 This paper considers static neural networks (systems without memory) formed by interconnections without loops, called feedforward networks.2 In addition, this work considers a backpropagation learning scheme with adaptation of learning rate. There have been many additions to, or adaptations of, the basic backpropagation scheme, such as adaptive learning rate, learning with momentum, and the like,1'2 for improving convergence of the learning scheme, and these are not addressed in this paper. This paper explores a modification of the backpropagation learning scheme using information on the gradient of the target outputs with respect to the inputs of the network. The performance of the modified learning scheme is compared with that of the standard backpropagation learning scheme.2 A feedforward network with a single hidden layer trained with this modified learning scheme is applied in structural optimization problems. Figure 1 shows a feedforward net with a single hidden layer. The restriction to a single hidden layer is not a limitation of the modified learning scheme, and the new training scheme can be used on networks with multiple hidden layers also. The use of neural networks for structural response approximation has been reported by several authors.46 The primary reason for investigating such properties is to assess the applicability of trained neural networks for quick and accurate approximation of response functions during structural design optimization. Structural optimization, utilizing either gradient-based numerical search or genetic search algorithms, requires repeated function (response) evaluations during the search process. A single finite element analysis of large-scale structures requires a significant amount of computational time and resource and hence the need for robust approxima

The present status of the constants and verification of the laws of thermal radiation of a uniformly heated enclosure

IntroductionSimulating thermal convection in Earth and planetary interiors is crucial for various applications, including benchmarking numerical codes (e.g. [1], [2]), deriving scaling laws for heat transfer in complex flows (e.g. [3], [4]), determining mixing efficiency (e.g. [5], [6]) and the characteristic spatial wavelengths of convection (e.g. [7]). However, reaching a statistically-steady state in these simulations, even in 2D, can be computationally expensive. While choosing "close-enough" initial conditions can significantly speed simulations up, this selection process can be challenging, especially for systems with multiple controlling parameters. This work explores how machine learning can be leveraged to identify optimal initial conditions, ultimately accelerating numerical convection simulations on their path to a statistically-steady state.Convection model and neural network architectureWe have compiled a comprehensive dataset comprising 128 simulations of Rayleigh-Bénard convection within a rectangular domain of aspect ratio 4, characterized by variable viscosity and internal heating. A randomized parameter sweep was employed to vary three key factors: the ratio of internal heat generation to thermal driving forces (RaQ/Ra), the temperature-dependent viscosity contrast (FKT), and the pressure-dependent viscosity contrast (FKV). Each simulation was executed until the root mean square velocity and mean temperature attained a statistically stationary state. Subsequently, we extracted the time-averaged one-dimensional temperature profile, horizontally averaged across the domain, over the final 200 time steps of each simulation.Figure 1: Schematic of the neural network used to predict statistically-steady temperature profiles.We used a feedforward neural network (NN) to predict the 1D temperature profiles as a function of the control parameters (Fig. 1). We modified the formulation of the NN from the one we used in [8] in the following ways. First, instead of each training example consisting of the full temperature profile, we now predict temperature at a given spatial point in depth (y). This allows the network to learn and predict profiles on variable computational grids, making it more adaptable. This formulation also avoids the "wiggles" encountered when predicting high-dimensional profiles. Second, we use skip-connections. For a given hidden layer, the features from all the previous hidden layers are added to it before applying the activation function. Third, we use SELU instead of tanh for faster convergence. Fourth, we concatenate the parameters to the last hidden layer, a technique shown to improve learning in some cases  (e.g., [9]). Fifth, as a consequence of conditioning the NN on spatial points, we augment the data with copies of points from the bottom and top thermal boundary layers to ensure that each batch is likely to be more representative of relevant temperature features occurring at the outer boundaries of the domain. We split the dataset into training, cross-validation, and test sets with 96, 15, and 16 simulations, respectively. To test the extrapolation capacity of the network, we pick simulations for the test set only where at least one of the simulation parameters exceeds a certain threshold. The cross-validation dataset consists of interpolated values only and is used additionally to choose the best performing NN architecture and save network weights only when the cross-validation loss improves. ResultsWe test the NN against the following baselines: linear regression, kernel ridge regression, and nearest neighbor. As shown in Table 1, when predicting unseen profiles from interpolated cases in the cross-validation set or even the extrapolated cases (Test set), the NN predictions are the most accurate among the tested possibilities. We visualize some of the profiles predicted by these algorithms in the cross-validation set in Fig. 2. Linear regression results are left out for ease of visualization.  Algorithm                  Training set Cross-validation set Test set Linear Regression  0.0385 0.0388 0.0676 Kernel Ridge Regression  0.0148 0.0147 0.0371 Neural Network 0.0071  0.0071  0.0187 Nearest Neighbor   0.0 0.0282  0.0495 Table 1: Mean absolute error of the prediction of 1D temperature profiles of statistically-steady simulations based on different tested algorithms. The lowest error per dataset is highlighted in bold.The plots in Fig. 2 corroborate the obtained statistics about the accuracy of different methods. When the simulation parameters we are predicting happen to be close to a simulation in the training set, the nearest neighbor profile already provides a good estimate. Otherwise, the NN predictions appear to be the most accurate. This is encouraging because it shows that even for small datasets such as the one we considered, machine learning can already start delivering useful results. Figure 2: Four temperature profiles from the cross-validation dataset (solid red lines) and the corresponding predictions from different models (dashed lines).OutlookWe are currently exploring how initial conditions impact the time a system takes to reach a statistically-steady state in terms of temperature and RMS velocity. We're comparing four starting points: hot, cold, linear temperature profiles, and predictions from our neural network. This will quantify the efficiency gain achieved by using the NN predictions to accelerate simulations towards equilibrium. In addition, we also plan to share the dataset and trained neural network with the community.

This paper proposes a wafer alignment method using a feedforward neural network (FNN). A wafer is placed on a vacuum table and its misalignment is inspected by a frame grabber. The alignment of the wafer is repeatedly corrected by adjusting its kinematic positions until the misalignment becomes zero. The training set is composed of the misalignment and the <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">xy</i> ¿ compensation. Fifteen sets of misalignment data were measured by the conventional alignment method. The FNN has four normalized inputs as mark locations in the field of vision and three normalized outputs for the kinematic compensation. The macro and micro steps share the FNN by means of a multiplexer (MUX) and a scaler. The training rule was back-propagation (BP). The proposed method was applied to a wafer manufacturing machine and its alignment performance was compared with that of conventional methods. The alignment time was reduced by the FNN model which was especially efficient in the case of macroscopic alignment.

Protein-based therapeutics are playing an increasingly important role in the treatment of diseases, including diabetes and cancer. The viability of these treatments, however, are highly dependent on the stability of the therapeutic, since stability affects both the shelf life of the therapeutic as well as its active life in the body. Stability engineering can, therefore, be used to increase the effectiveness of protein-based therapeutics. Computational methods of protein stability prediction have been under development for about a decade, but complex molecular interactions make stability prediction difficult and computationally intensive. A rapid computational method of protein stability prediction is developed using feed-forward neural networks and used to predict mutation-induced stability changes in Staphylococcal nuclease. The input to the neural network consisted of sequences of evolutionarily based amino acid similarity scores that were obtained through the comparison of the amino acids in a mutation containing sequence to their positional counterparts in the baseline wild-type amino acid sequence. A training set was created which consisted of similarity score sequences, for which the stabilities of the corresponding amino acid sequences were known, paired with the relative stabilities of the sequences to that of the baseline. Back-propagation of error was used to train the network to output accurate relative stability scores for the sequences in the training set. Neural network-based relative stability predictions for 55 sequences containing mutation combinations not found in the training set had an accuracy of 92.8%.

Three main parameters characterize the efficiency of algorithms that solve the Consensus Problem. The ratio between the total number of processors and the maximum number of faulty processors (n and t, respectively), the number of rounds, and the size of any single message. Lower bounds exist for each one of the three. In this paper we present two families of algorithms, each achieving the lower bound for one parameter and a trade-off between the other two. The first family includes algorithms where, given an integer k, the algorithm always requires the minimal possible number of rounds (t+1), with n=k(3t+1) processors and messages of size at most t O(t/k). To the second family belong algorithms in which all messages are of one bit size, the number of processors is t O((k+1)/k) and the number of rounds is t+t O ((k−1)/k). These two families are based on a very simple algorithm with (2t+1)(t+1) processors using the minimal number of rounds and the minimal message size (one bit).

In this paper, we address the problem of random field estimation with wireless sensor networks in the absence of CSI at the sensor nodes. We assume that sensors encode their observations at a common and constant rate which, on the one hand, determines the quantization noise at the encoders and, on the other, the outage probability experienced in the sensor-to-fusion center fading channels. These effects, arising from the power and bandwidth constraints that we impose, along with the spatial sampling process itself result into some distortion that we attempt to minimize by carefully selecting the optimal number of sensor nodes and the aforementioned encoding rate. On the basis of a (tight) upper bound, we derive closed-form expressions of the corresponding optimal values. Computer simulation results are mostly given in terms of optimal encoding rate and network size for different operating conditions (SNR, spatial variability of the random field) and reconstruction distortion vs. network size.

We present an algorithm for regular, personalized all- to- all communication, in which every processor has an individual message to deliver to every other processor. Our machine model is a cluster of processing nodes where each node, possibly consisting of several processors, can participate in only one communication operation with another node at a time. The nodes may have different numbers of processors. This general model is important for the implementation of all-to-all communication in libraries such as MPI where collective communication may take place over arbitrary subsets of processors. The algorithm is optimal up to an additive term that is small if the total number of processors is large compared to the maximal number of processors in a node.

An increasing number of applications of dynamic neural networks has been developed for digital signal processing (DSP), dynamic neural networks are feedforward neural networks with commonly used scalar synapses replaced by linear filters. This provides feedforward neural networks with the capability of performing dynamic mappings, which depend on past input values, dynamic neural networks are suitable for time series prediction, nonlinear system identification, and signal processing applications. Their most popular types are Finite Impulse Response (FIR) neural networks, which are obtained by replacing synapses with finite impulse response filters. Due to their guaranteed stability characteristic and easy to minimize error surface they have been used with great success in many applications such as signal enhancement, noise cancellation, classification of input patterns, system identification, prediction, and control.. Most of the works on system identification using neural networks are based on multilayer feedforward neural networks with backpropagation learning or more efficient variations of this algorithm, an elegant method for training layered networks. This paper is based on work in a Dynamic System Modeling (DYSMO) and as an application for speed control of DC motor drive.

In order to provide an arbitrary and fully dynamic connectivity in a network of processors, transport mechanisms must be implemented, which provide the propagation of data from processor to processor, based on addresses contained within a packet of data. Such data transport mechanisms must satisfy a number of requirements, namely deadlock and livelock freedom, good hot-spot performance, high throughput and low latency. The paper proposes a solution to these problems, which allows deadlock free, adaptive, high throughput packet routing to be implemented on networks of processors. Examples are given which illustrate the technique for two-dimensional (2D) array and toroidal networks. The implementation of this scheme on arrays of transputers is considered. The scheme also serves as a basis for very-low latency routing strategy introduced here as well.

In order to provide an arbitrary and fully dynamic connectivity in a network of processors, transport mechanisms must be implemented, which provide the propagation of data from processor to processor, based on addresses contained within a packet of data. Such data transport mechanisms must satisfy a number of requirements - deadlock and livelock freedom, good hot-spot performance, high throughput and low latency. This paper proposes a solution to these problems, which allows deadlock free, adaptive, high throughput packet routing to be implemented on networks of processors. Examples are given which illustrate the technique for 2-D array and toroidal networks. An implementation of this scheme on arrays of transputers is described. The scheme also serves as a basis for a very low latency routing strategy named the mad postman , a detailed implementation of which is described here as well.

The problem of optimal scheduling n tasks in a parallel processor system is studied. The tasks are malleable, i.e., a task may be executed by several processors simultaneously and the processing speed of a task is a nonlinear function of the number of processors allocated to it. The total number of processors is m and it is an upper bound on the number of processors that can be used by all the tasks simultaneously. It is assumed that the number of processors is sufficient to process all the tasks simultaneously, i.e. n≤m. The objective is to find a task schedule and a processor allocation such that the overall task completion time, i.e. the makespan, is minimized. The problem is motivated by real-life applications of parallel computer systems in scientific computing of highly parallelizable tasks. An O(n) algorithm is presented to solve this problem when all the processing speed functions are convex. If these functions are all concave and the number of tasks is a constant, the problem can be solved in polynomial time. A relaxed problem, in which the number of processors allocated to each task is not required to be integer, can be solved in O(nmax {m,nlog 2 m}) time. It is proved that the minimum makespan values for the original and relaxed problems coincide. For n=2 or n=3, an optimal solution for the relaxed problem can be converted into an optimal solution for the original problem in a constant time.

We propose a self-learning method for automatic wafer alignment in the semiconductor manufacturing process. A feed forward neural network is trained by and used for wafer alignment. The network determines the movement of kinematic parts from the misalignment inspected by machine vision. However, it is time-consuming and inconvenient to obtain training data in this way. So, we built an automatic learning rule to gather the data and train the network. The network may determine wrong outputs and cause other misalignments at first, but the error can decrease as the training proceeds. The training sets consisted of a variation of misalignment data and the movement of an alignment stage. Five recent sets are used for training and others are dismissed or forgotten. This retrained network tried aligning, measured misalignment, and made new training sets. This sequence makes it possible to acquire alignment skill and automate the process. After learning, automatic alignment accomplished sub-pixel accuracy for several cases of misalignment. The result showed that the proposed method could be applied to the semiconductor manufacturing process. Its performance improved about 6% compared with conventional algorithms.

Reclaimed asphalt pavement (RAP) is one of the waste materials that highway agencies promote to use in new construction or rehabilitation of highways pavement. Since the use of RAP can affect the resilient modulus and other structural properties of flexible pavement layers, this paper aims to employ two different artificial neural network (ANN) models for modeling and evaluating the effects of different percentages of RAP on resilient modulus of hot-mix asphalt (HMA). In this research, 216 resilient modulus tests were conducted for establishing the experimental dataset. Input variables for predicting resilient modulus were temperature, penetration grade of asphalt binder, loading frequency, change of asphalt binder content compared to optimum asphalt binder content and percentage of RAP. Results of modeling using feed-forward neural network (FFNN) and generalized regression neural network (GRNN) model were compared with the measured resilient modulus using two statistical indicators. Results showed that for FFNN model, the coefficient of determination between observed and predicted values of resilient modulus for training and testing sets were 0.993 and 0.981, respectively. These two values were 0.999 and 0.967 in case of GRNN. So, according to comparison of R2 for testing set, the accuracy of FFNN method was superior to GRNN method. Tests results and artificial neural network analysis showed that the temperature was the most effective parameter on the resilient modulus of HMA containing RAP materials. In addition by increasing RAP content, the resilient modulus of HMA increased.

Introduction Image-based deep learning models (DLMs) have shown promise in predicting surgical complexity and wound complications for AWR with the potential to augment other clinical information in surgical decision-making. The purpose of this study is to evaluate a preoperative image-based DLM in predicting hernia recurrence. Methods A prospective single-institution database was queried for open AWR patients with preoperative CT imaging. Patient demographics and hernia recurrence were collected. Patient data were batched into training (80%) and test (20%) sets. A convolutional neural network was trained to evaluate image characteristics alone. A feedforward neural network (FNN) was also trained to evaluate clinical data alone. Another FNN was designed to incorporate the outputs from both models. The primary outcome was the ability to accurately predict hernia recurrence, as assessed by the area under the curve (AUC) for each model. Results There were 190 patients in this dataset who underwent AWR. Overall, 28 (14.7%) had a recurrence with a median follow up of 85 [56, 113] months. Preoperative CT imaging alone returned an AUC = 0.500. Clinical data alone performed better with an AUC = 0.667. Incorporating both models resulted in an AUC = 0.604. Conclusions In this study, the DLM with clinical data alone outperformed an image-based DLM in predicting recurrence. Clinical data alone also outperformed the model incorporating both clinical data and preoperative imaging. DLMs have shown significant potential to predict AWR outcomes and may be used in the future to guide care and aid in patient consent for surgery.