Historical database training method for neural networks

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

Neural network, a nonalgorithmic, nondigital, intensely parallel and distributive information processing system, is being used more and more everyday. The main interest in neural networks is rooted in the recognition that the human brain processes information in a different manner than conventional digital computers. Computers are extremely fast and precise at executing sequences of instructions that have been formulated for them. A human information processing system is composed of neurons switching at speeds about a million times slower than computer gates. Yet, humans are more efficient than computers at such computationally complex tasks as speech and other pattern-recognition problems. Artificial neural systems, or neural networks, are physical cellular systems that can acquire, store, and use experiential knowledge. The knowledge is in the form of stable states or mapping embedded in networks that can be recalled in response to the presentation of cues. In a typical neural data processing procedure, the database is divided into two separate portions called training and test sets. The training set is used to develop the desired network. In this process (depending on the paradigm that is being used), the desired output in the training set is used to help the network learn by adjusting the weights between its neurons or processing elements. Once the network has learned the information in the training set and has"converged," the test set is applied to the network for verification. It is important to note that although the user has the desired output of the test set, it has not been seen by the network. This ensures the integrity and robustness of the trained network. A handful of articles on the use of neural networks in the petroleum industry has appeared in SPE conferences, proceedings, and publications in the past 2 years. These articles can be divided into two categories: those that use neural networks to analyze formation lithology from well logs and those that use neural networks to pick a reservoir model to be used in conventional welltest interpretation studies. These tasks are usually done by log analysts and reservoir engineers, and their automation using a fault-tolerant process may prove valuable. Neural networks can help engineers and researchers by addressing some fundamental petroleum engineering problems as well as specific ones that conventional computing has been unable to solve. Petroleum engineers may benefit from neural networks on occasions when engineering data for design and interpretations are less than adequate. This is an especially common occurrence in the Appalachian basin, where some fields are quite old. Lack of adequate engineering data may also be encountered because of the high cost of coring, well testing, and so on. Neural networks have shown great potential for generating accurate analysis and results from large amounts of historical data that otherwise would seem not to be useful or relevant in the analysis. An example of such a problem was encountered by a gas company for a gas storage field in Ohio. In the absence of appropriate data, which normally would make engineering design and evaluation of the fracturing jobs virtually impossible, a carefully designed neural network was able to predict the performance of fracturing jobs with great accuracy. A linear plot of the actual fracturing job results (data never seen by the network during training) and network predictions resulted in a correlation coefficient of 0.98, where 1.00 is a perfect match. Neural networks have proved to be valuable pattern-recognition tools. They are capable of finding highly complex patterns within large amounts of data. A relevant example is well log interpretation. It is generally accepted that there is more information embedded in well logs than meets the eye. Determination, prediction, or estimation of formation permeability without actual laboratory measurement of the cores or interruption in production for well test data collection has been a fundamental problem for petroleum engineers. From geophysical well log data, it was possible to predict and/or estimate permeability of a highly heterogeneous formation in West Virginia, as shown in Fig 1. Reference Mohaghegh, S., Arefi, R., and Ameri, S.: "Design and Development of an Artificial Neural Network for Prediction of Formation Permeability,"paper SPE 28237 presented at the 1994 SPE Petroleum Computer Conference, July31-Aug. 3, Dallas.

A new classifier neural network is described for distortion-invariant multi-class pattern recognition. Its input training data in different classes are described by a feature space. As a distortion parameter (such as aspect view) of a training set object is varied, an ordered training set is produced. This ordered training set describes the object as a trajectory in feature space, with different points along the trajectory corresponding to different aspect views. Different object classes are described by different trajectories. Classification involves calculation of the distance from an input feature space point to the nearest trajectory (this denotes the object class) and the position of the nearest point along that trajectory (this denotes the pose of the object). Comparison to other neural networks and other classifiers show that this feature space trajectory neural network yields better classification performance and can reject non-object data. The FST classifier performs well with different numbers of training images and hidden layer neurons and also generalizes better than other classifiers.

MACHINE LEARNING PREDICTION OF IN-HOSPITAL DISEASE PROGRESSION IN COMMUNITY-ACQUIRED PNEUMONIA: DERIVATION AND VALIDATION OF CLINICAL DETERIORATION IN PNEUMONIA (CDIP)

The paper presents a novel architecture and method for training neural networks to produce synthesized speech in a particular voice and speaking style, based on a small quantity of target speaker/style training data. The method is based on neural network embedding, i.e. mapping of discrete variables into continuous vectors in a low-dimensional space, which has been shown to be a very successful universal deep learning technique. In this particular case, different speaker/style combinations are mapped into different points in a low-dimensional space, which enables the network to capture the similarities and differences between speakers and speaking styles more efficiently. The initial model from which speaker/style adaptation was carried out was a multi-speaker/multi-style model based on 8.5 hours of American English speech data which corresponds to 16 different speaker/style combinations. The results of the experiments show that both versions of the obtained system, one using 10 minutes and the other as little as 30 seconds of target data, outperform the state of the art in parametric speaker/style-dependent speech synthesis. This opens a wide range of application of speaker/style dependent speech synthesis based on small quantities of training data, in domains ranging from customer interaction in call centers to robot-assisted medical therapy.

This focus issue of Physiological Measurement follows the 38th Annual International Computing in Cardiology (CinC) Conference, hosted in Hangzhou, China in September 2011 by Zhejiang University. Each year, the NIH-sponsored PhysioNet resource (http://physionet.org/) runs an open competition lasting several months, aimed at encouraging the development of solutions to an unsolved or poorly solved problem in biomedicine, in most cases making use of relevant clinical and experimental data provided freely by PhysioNet. Participants in these annual challenges discuss their diverse approaches to the Challenge problems during dedicated scientific sessions at CinC. The topics of these PhysioNet/CinC Challenges range from physiologic signal processing and analysis to forecasting and modelling clinically important events and processes.

A neural network approach has been applied to model downhole resistivity tools, i.e., to generate a synthetic tool response for a given earth resistivity model. The microlaterolog (MLL), shallow dual laterolog (DLLs), and deep dual laterolog (DLLd) tools are modeled using neural networks to demonstrate this approach. Efforts have been made to select various neural network parameters, including the type of neural network, the length of input data for training, the number of hidden nodes, and the number of training samples. A modular neural network (MNN) has been selected because it can facilitate the training and prediction of tool responses in formations with large resistivity variations. The input data for training are taken to be the model formation resistivity values sampled over a depth window. The window length is chosen based on the tool lengths. Three different window lengths are used for experiments: 6.1, 9.1, and 30.5 m. We found the longer window lengths generally have higher modeling accuracy for the three different types of logging tools. The number of hidden nodes needed to yield satisfactory training and prediction data varies from 8 to 25, depending on the type of tool and the window length. Up to 30 000 training samples have been collected to train the MNN. Our modeling examples show that the trained MNN can achieve about 90% accuracy for the MLL log response and about 83% accuracy for the DLLs and DLLd responses. The modeling errors can be described roughly with a Gaussian distribution.

In this work, the right training criterion for optimal prediction of the desired output by the neural network model was our focus. The application of the early stopping technique in the neural network training of radial basis function model in comparison to the same model neural network training without application of early stopping technique was the basis for the work and the work validates the importance of early stopping technique for good network generalization through prevention of overfitting. An over-fitted radial basis function neural network model possesses the tendency for dataset memorization including noise and thus, poorly learns the dataset pattern. From the training results, the application of an early stopping technique inclusive of other vital hyper-parameters during the training of the radial basis function model demonstrates a measure that is very imperative in the prevention of over-fitting during network training. The training results show that the radial basis function model training without the application of an early stopping technique displayed a “learning and slow” phenomenon where the model stopped learning the dataset at a stage without optimum prediction of the dataset pattern. This phenomenon was evaded by the application of an early stopping technique at the ratio of 70%:15%:15% for training, testing, and validation of the standardized dataset. The radial basis function model neural network training regression results show optimum prediction results of R=0.99420 with a training set of R=0.9985 and a test set of R=0.9899 showing very close prediction of the training set by the test set on training with early stopping technique in comparison to the model training without the application of early stopping technique where R= 0.97200 with a training set of R= 0.9840 and a test set of R=0.9612 showing very wide prediction of the training set by the test set. Overfitting often results in good model training but poor test error as the network model learns the wrong structure in the training data or stops learning at a point as seen from the work. This was optimally evaded on the application of the early stopping technique in the work for robust prediction of signal power loss using a long-term evolution dataset.

IntroductionSimulating thermal convection in Earth and planetary interiors is crucial for various applications, including benchmarking numerical codes (e.g. [1], [2]), deriving scaling laws for heat transfer in complex flows (e.g. [3], [4]), determining mixing efficiency (e.g. [5], [6]) and the characteristic spatial wavelengths of convection (e.g. [7]). However, reaching a statistically-steady state in these simulations, even in 2D, can be computationally expensive. While choosing "close-enough" initial conditions can significantly speed simulations up, this selection process can be challenging, especially for systems with multiple controlling parameters. This work explores how machine learning can be leveraged to identify optimal initial conditions, ultimately accelerating numerical convection simulations on their path to a statistically-steady state.Convection model and neural network architectureWe have compiled a comprehensive dataset comprising 128 simulations of Rayleigh-Bénard convection within a rectangular domain of aspect ratio 4, characterized by variable viscosity and internal heating. A randomized parameter sweep was employed to vary three key factors: the ratio of internal heat generation to thermal driving forces (RaQ/Ra), the temperature-dependent viscosity contrast (FKT), and the pressure-dependent viscosity contrast (FKV). Each simulation was executed until the root mean square velocity and mean temperature attained a statistically stationary state. Subsequently, we extracted the time-averaged one-dimensional temperature profile, horizontally averaged across the domain, over the final 200 time steps of each simulation.Figure 1: Schematic of the neural network used to predict statistically-steady temperature profiles.We used a feedforward neural network (NN) to predict the 1D temperature profiles as a function of the control parameters (Fig. 1). We modified the formulation of the NN from the one we used in [8] in the following ways. First, instead of each training example consisting of the full temperature profile, we now predict temperature at a given spatial point in depth (y). This allows the network to learn and predict profiles on variable computational grids, making it more adaptable. This formulation also avoids the "wiggles" encountered when predicting high-dimensional profiles. Second, we use skip-connections. For a given hidden layer, the features from all the previous hidden layers are added to it before applying the activation function. Third, we use SELU instead of tanh for faster convergence. Fourth, we concatenate the parameters to the last hidden layer, a technique shown to improve learning in some cases  (e.g., [9]). Fifth, as a consequence of conditioning the NN on spatial points, we augment the data with copies of points from the bottom and top thermal boundary layers to ensure that each batch is likely to be more representative of relevant temperature features occurring at the outer boundaries of the domain. We split the dataset into training, cross-validation, and test sets with 96, 15, and 16 simulations, respectively. To test the extrapolation capacity of the network, we pick simulations for the test set only where at least one of the simulation parameters exceeds a certain threshold. The cross-validation dataset consists of interpolated values only and is used additionally to choose the best performing NN architecture and save network weights only when the cross-validation loss improves. ResultsWe test the NN against the following baselines: linear regression, kernel ridge regression, and nearest neighbor. As shown in Table 1, when predicting unseen profiles from interpolated cases in the cross-validation set or even the extrapolated cases (Test set), the NN predictions are the most accurate among the tested possibilities. We visualize some of the profiles predicted by these algorithms in the cross-validation set in Fig. 2. Linear regression results are left out for ease of visualization.  Algorithm                  Training set Cross-validation set Test set Linear Regression  0.0385 0.0388 0.0676 Kernel Ridge Regression  0.0148 0.0147 0.0371 Neural Network 0.0071  0.0071  0.0187 Nearest Neighbor   0.0 0.0282  0.0495 Table 1: Mean absolute error of the prediction of 1D temperature profiles of statistically-steady simulations based on different tested algorithms. The lowest error per dataset is highlighted in bold.The plots in Fig. 2 corroborate the obtained statistics about the accuracy of different methods. When the simulation parameters we are predicting happen to be close to a simulation in the training set, the nearest neighbor profile already provides a good estimate. Otherwise, the NN predictions appear to be the most accurate. This is encouraging because it shows that even for small datasets such as the one we considered, machine learning can already start delivering useful results. Figure 2: Four temperature profiles from the cross-validation dataset (solid red lines) and the corresponding predictions from different models (dashed lines).OutlookWe are currently exploring how initial conditions impact the time a system takes to reach a statistically-steady state in terms of temperature and RMS velocity. We're comparing four starting points: hot, cold, linear temperature profiles, and predictions from our neural network. This will quantify the efficiency gain achieved by using the NN predictions to accelerate simulations towards equilibrium. In addition, we also plan to share the dataset and trained neural network with the community.

Machine learning (ML) is the most common technique for predicting the future or for classifying information, to help people make the required decisions. ML techniques are based on algorithms – sets of mathematical procedures that explain the relationship between variables. ML algorithms are trained in situations where they learn from past data and can even evaluate historical data. After extensive training, the algorithm can recognize patterns sufficiently to make predictions. ML provides methods, techniques, and tools that can help in solving diagnostic and prognostic problems in a variety of medical domains. It is being used for the analysis of the importance of clinical parameters and their combinations for prognosis, e.g., for prediction of disease progression, for the extraction of medical information to predict research outcomes, for therapy planning and support, and for overall patient management. ML is also being used for data analysis, such as detection of patterns in the data by dealing appropriately with imperfect data, interpretation of continuous data generated in the Intensive Care Unit, and for intelligent alarms, resulting in effective and efficient monitoring. Successful implementation of ML methods can help the integration of computer-based systems into the healthcare environment, providing opportunities to facilitate and enhance the work of medical experts, and ultimately to improve the efficiency and quality of medical care. Most ML methods can be categorized into one of two types of learning techniques, namely supervised or unsupervised algorithms. A supervised learning algorithm takes a known set of input data (the training set) and known responses to the data (the output) and trains a model to generate reasonable predictions for the response to new input data. In medicine, supervised learning involves training a model to relate a person’s characteristics (e.g., height, weight, smoking status) to a certain outcome (onset of diabetes within five years, for example). Once the algorithm is successfully trained, it will be capable of making outcome predictions when supplied with new data. Predictions which are made by models trained using supervised learning can be either discrete (e.g., positive or negative, benign or malignant) or continuous (e.g., a score from 0 to 100). A supervised model which produces discrete categories (sometimes referred to as classes) is referred to as a classification algorithm. Examples of classification algorithms include those which predict whether a tumor is benign or malignant, or to establish whether comments written by a patient convey a positive or negative sentiment. In practice, classification algorithms return the probability of a class (between 0 for impossible and 1 for certain). Common classification algorithms are Logistic Regression, K-Nearest Neighbor, Support Vector Machine, Neural Network, and Naïve Bayes. A supervised model which returns a prediction of a continuous value is known as a regression algorithm, which might be used by ML to predict an individual’s life expectancy or the tolerable dose of chemotherapy. Supervised ML algorithms are typically developed using a dataset, which contains several variables and a relevant outcome. In contrast with supervised learning, unsupervised learning does not involve a predefined outcome. In unsupervised learning, patterns are sought by algorithms without any input from the user. Unsupervised techniques are thus exploratory and used to find undefined patterns or clusters which occur within datasets. In this chapter, the use of different types of supervised and unsupervised ML algorithms for various applications in health care are discussed.

The Sixth International Symposium on Neural Networks (ISNN 2009)

The calculations of the load of an unstable two-port network were made. Traditional calculation methods involve laborious parameter redefinition and recalculation of the desired values. In contrast, a neural network includes possible changes in the two-port parameters in the training data. The feedforward neural network training data represents a set of load values and corresponding input current values. Such data are calculated using some change steps of the load and two-port parameters. The training data are split into training, validation, and test sets. It was established that in the training epochs, the neural network reveals that internal pattern in those three sets. Therefore, small errors are obtained. However, the errors appear for the extended control data in different step types. Combining training data with regular and irregular step change parameters eliminates this pattern, so the network shows the capability to generalize. A probability index of the quantification of training quality yields a trade-off between the size of the training data, the accuracy obtained, and the number of neurons. Because an unstable two-port has three parameters, unsatisfactory results were obtained using three base load values. In turn, four excessive base loads radically increase the precision and capability generalization of the network. The established features of the behavior of the neural network provide a base for solving practical "streaming" tasks of different physical nature based on known analogies.

ADANOISE: Training neural networks with adaptive noise for imbalanced data classification

Effective training data selection in tool condition monitoring system

Gastroesophageal Reflux Disease (GERD) is a disorder in which the contents of the stomach reflux repeatedly into the esophagus (esophagus), which causes symptoms and or complications that, interfere. The purpose of this study was to determine the level of risk of GERD in respondents based on diet, lifestyle, and perceived symptoms. The method used in this study is a decision tree with the C.45 algorithm. To make a decision tree WEKA tools are used, where data obtained from the results of a questionnaire distributed to 51 respondents to determine diet and lifestyle, then to predict the risk level of GERD a web-based decision support system was built using the Php programming language and the MySQL database. The input needs consist of respondent data input, questionnaire question data input, news data input, and training data input, with process requirements that include respondent data processing, questionnaire question data processing, news data processing, questionnaire data processing, and reporting process, which will produce training data output reports by gender, training data reports by date and prediction results. This study resulted in a design of a decision support system to determine the risk level of GERD that can help medical staff aware and educate the public to recognize early symptoms of GERD disease and to help general practitioners advise patients with GERD.