Question classification in Persian using word vectors and frequencies

The Sixth International Symposium on Neural Networks (ISNN 2009)

Marker-based Optical Motion Capture (OMC) systems and associated musculoskeletal (MSK) modelling predictions offer non-invasively obtainable insights into muscle and joint loading at an in vivo level, aiding clinical decision-making. However, an OMC system is lab-based, expensive, and requires a line of sight. Inertial Motion Capture (IMC) techniques are widely-used alternatives, which are portable, user-friendly, and relatively low-cost, although with lesser accuracy. Irrespective of the choice of motion capture technique, one typically uses an MSK model to obtain the kinematic and kinetic outputs, which is a computationally expensive tool increasingly well approximated by machine learning (ML) methods. Here, an ML approach is presented that maps experimentally recorded IMC input data to the human upper-extremity MSK model outputs computed from ('gold standard') OMC input data. Essentially, this proof-of-concept study aims to predict higher-quality MSK outputs from the much easier-to-obtain IMC data. We use OMC and IMC data simultaneously collected for the same subjects to train different ML architectures that predict OMC-driven MSK outputs from IMC measurements. In particular, we employed various neural network (NN) architectures, such as Feed-Forward Neural Networks (FFNNs) and Recurrent Neural Networks (RNNs) (vanilla, Long Short-Term Memory, and Gated Recurrent Unit) and a comprehensive search for the best-fit model in the hyperparameters space in both subject-exposed (SE) as well as subject-naive (SN) settings. We observed a comparable performance for both FFNN and RNN models, which have a high degree of agreement (ravg,SE,FFNN=0.90±0.19, ravg,SE,RNN=0.89±0.17, ravg,SN,FFNN=0.84±0.23, and ravg,SN,RNN=0.78±0.23) with the desired OMC-driven MSK estimates for held-out test data. The findings demonstrate that mapping IMC inputs to OMC-driven MSK outputs using ML models could be instrumental in transitioning MSK modelling from 'lab to field'.

Current state-of-the-art neural network-based Question Answering (QA) systems consist of both Recurrent Neural Networks (RNNs) and Feedforward Neural Networks (FFNNs). They generally performed well on 19 of the 20 tasks in the benchmark bAbI dataset. The only task that they failed badly is a task involving inductive reasoning where the order of the facts is not important in producing the correct answer. In this paper, we removed the RNNs from DMN+ QA system to form the ff-DMN system. The results demonstrate that ff-DMN improves the accuracy of the induction task significantly. Further experiments reveal that using RNNs is important if intra-sentence reasoning is required while it may adversely affect the performance if inter-sentence reasoning is involved. Finally, by incorporating ff-DMN and DMN+ our ensemble model outperforms the other QA systems on all the 20 tasks.

Intrusion detection is a major challenge for security experts in the cyber world. Traditional IDS failed to detect complex and unknown cyber-attacks. Machine learning has become a vibrant technology for cybersecurity. There exists several machine learning algorithms to detect intrusion. Most classifiers are well suited to detect the attacks. However, improving accuracy and detecting unknown attacks in existing IDSs is a great challenge. Therefore, the detailed comparative study of various machine learning approaches such as artificial neural networks, support vector machine, decision tree, and hybrid classifiers used by researchers for intrusion detection are done. Deep learning is an emerging approach which suits well for large data. Deep learning techniques find optimal feature set and classify low-frequency attacks better than other techniques. This study also summarizes literatures in deep learning approaches such as deep auto-encoder, Boltzmann machine, recurrent neural networks, convolutional neural networks, and deep neural networks. Moreover, the datasets used in various literatures and the analysis of deep learning approaches based on the performance metrics are also done. Future directions to detect intrusion are also provided. This study in fact will be helpful to develop IDS based on artificial intelligence approaches such as machine learning and deep learning.KeywordsMachine learningIntrusion detectionFeature selectionDeep learningCyber securityClassifier

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

The adoption of wind energy has grown significantly in recent years. New, cost-effective technologies have been developed, led by customer awareness of green technologies and a legal framework proposed at the European Union level. The stochastic nature of wind speed is transferred to wind turbine output, making wind energy difficult to predict. The main scope of predicting wind energy production is to be proactive in balancing and reserving energy to meet demand. When the prediction identifies a potential gap between supply and demand, additional energy from other sources must be generated and supplied. Creating a synergy of physical devices through advanced sensing capabilities, software, storage and analytics capabilities, the Industrial Internet of Things is enabling the effective transition to wind energy through automation by removing many of the disadvantages in a way that has recently become accessible. This research focuses on the data analytics, proposing a fast univariate network-based approach for wind energy prediction, using Feed Forward Neural Networks, Recurrent Neural Networks, Long-Short Term Memory, Gated Recurrent Unit, and Convolutional Neural Networks. Moreover, by introducing the theoretical fundamentals, the implementation method and the hyperparameters of the final models, this article becomes unique in the context of wind energy. At the time of this study, no prior research studies have presented a direct comparison between feedforward, recurrent, and convolutional neural networks ? these being the most important in the field of supervised learning.

Arabic language can be used by native and non- native speakers; due to Arabic is the language of the holy book of Muslims. In this paper, Arabic phoneme recognition system is proposed based on Malay speakers. This system consists of three main stages. The first stage is noise reduction and it aims to enhance the phoneme signals by excluding the unvoiced signals and keep only the voiced signal. Wiener filter is adapted to accomplish this task. The second stage is based on Mel-Frequency Cepstral Coefficients method to extract a vector of features to represent each phoneme signal. Eventually, pattern recognition neural network is designed as recognizer. The proposed system produces sufficient outcomes with 20 hidden neurons. Keywords—Arabic; Malay; pattern recognition; Wiener; Mel- Frequency Cepstral Coefficients I. INTRODUCTION It is well known the wide range of Artificial Neural Networks (ANN) application in speech recognition, financial, telecommunications, electronics and medical. As for speech recognition system for Arabic language, (1) & (2) have addressed the implementation of recognition system based on Arabic native speakers. In general, neural networks (NNs) for phonemes recognition are divided into several categories for instance Very Large Scale Integration (VLSI) NN that can be divided into digital and analogue with digital NNs being more compatible with feed-forward neural networks while analogue NNs are found to be more successful with recurrent NNs (3). From Automatic Speech System (ASR) point of view, NNs for recognition purpose can be divided into two main category namely conventional neural networks such as Multilayer Perceptron (MLP) and Radial Basis Function (RBF) and secondly Recurrent Neural Networks (RNN). The first category of NNs was implemented as pattern classifiers and proven capable for recognition purpose but not at par as compared to Hidden Markov Model (HMM) performance (2). At present, HMM has been proven to be the most successful approach in ASR research area until recently the possibility to hybrid both HMM and ANN (4) & (5). In general, speech recognition systems consist of three stages specifically noise reduction, feature extraction and classification. The aim of noise reduction is to reduce the level of noise in the speech signal and make it more compatible for the ASR. As for feature extraction, this process attempted to extract the most relevant features related to the phoneme signal. The extracted features acted as input features of the NN during training and testing. Finally, the purpose of classification process is to arrange the most similar and related features in categories based on the training phase. Basically, networks training database are being observed and updated during training session of the networks, involving the weights and biases arguments. Networks with lower errors can be considered as better networks (6). Hence, this paper deems to explore further the ability of NN for Arabic phonemes recognition spoken by Malay individuals. This paper is arranged as following; related work is discussed in section II followed by section III that discussed method to be implemented based on Wiener filter, Mel-Frequency Cepstral Coefficients and pattern recognition neural network. Section IV elaborated experimental analysis and results and followed by Conclusion section.

Machine learning (ML) frameworks are investigated for use in simultaneous tracking and navigation (STAN) with low Earth orbit (LEO) satellites. STAN is a navigation paradigm that utilizes specialized LEO receivers to extract navigation observables (e.g., pseudorange and Doppler) from LEO satellite signals. Two neural network architectures are compared: Feed Forward Neural Network (FFNN) and Recurrent Neural Network (RNN). Additionally, two ML-based orbit determination frameworks are compared: ephemeris propagation and residual error propagation. The objective of the comparison is to select an approach with the lowest open-loop propagation error as well as computational cost. Based on simulation results, a nonlinear autoregressive neural network with exogenous inputs (NARX) embedded within the residual error modeling framework is selected as the best ML approach among the compared candidates. Experimental results are presented demonstrating a ground vehicle navigating for a total of 258 seconds, while receiving signals from two Orbcomm LEO satellites. Global navigation satellite system (GNSS) signals were artificially cut off for the last 30 seconds, during which the vehicle traversed a trajectory of 871 m. Two navigation frameworks are compared to estimate the vehicle’s trajectory: (i) LEO signal-aided inertial navigation system (INS) STAN framework using Simplified General Perturbation (SGP4) as its propagator and (ii) the proposed LEO signal-aided INS STAN framework using ML as its propagator. The STAN with SGP4 achieved a three-dimensional (3-D) position root-mean squared error (RMSE) of 30 m. In contrast, the proposed STAN with SGP4+NARX framework achieved a 3-D position RMSE of 3.6 m.

In the last decade, question classification is a strong signal for answer selection and help to find the structure of question sentences from sentences. For this paper, we evaluated the proposed pre-processing method for classifying the simple sentence to wh-question categories (What, When, Who, Where, and How) on Thai texts by considering Part-of-Speech tagging (POS). The performances are evaluated using classification accuracy obtained from traditional classification models including Naive Bayes, Logistic Regression, Support Vector Machine, K-Nearest Neighbors, and neural networks which employs Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN). We compared traditional models and neural networks; the experimental results showed that the result of the neural networks models better than the traditional model. The accuracy of the proposed model using the LSTM model with pre-trained word embedding is improved with an average F1 of 79.60%.

The frequency of armed conflicts increased during the last 20 years. The problems of the emergence of military disputes, not only concern social parameters, but also economic and financial dimensions. This study examines the potential impact of global geopolitical events on the stock market prices of the Dow Jones U.S. Aerospace & Defense Index and Foreign Exchange (FOREX) markets movements. We analyse whether defence stocks and exchange rate perform similarly during military incidents or geopolitical crises. We built an Autoregressive Moving Average Model with a Generalized Autoregressive Conditional Heteroskedasticity process (ARMA-GARCH) with the machine learning methods of Neural Networks, Deep Recurrent Convolutional Neural Networks, Deep Neural Decision Trees, Quantum Neural Networks, and Quantum Recurrent Neural Networks, aimed at detecting intraday patterns for forecasting defence stock market and FOREX markets disturbances in a market microstructure framework. The empirical results provide preliminary findings on the foreseeability of market disturbances and small differences are observed before and during geopolitical events. Additionally, we confirm the effectiveness of the hybrid model ARMA-GARCH with the machine learning approaches, being ARMA-GARCH-Quantum Recurrent Neural Network the technique that achieves the best accuracy results. Our work has a large potential impact on investment market agents and portfolio managers, as shocks from geopolitical events could provide a new methodology to support the decision-making process for trading in High-Frequency Trading.

Classification and prediction tasks are common in health research. With the increasing availability of vast health data repositories (e.g. electronic medical record databases) and advances in computing power, traditional statistical approaches are being augmented or replaced with machine learning (ML) approaches to classify and predict health outcomes. ML describes the automated process of identifying ("learning") patterns in data to perform tasks. Developing an ML model includes selecting between many ML models (e.g. decision trees, support vector machines, neural networks); model specifications such as hyperparameter tuning; and evaluation of model performance. This process is conducted repeatedly to find the model and corresponding specifications that optimize some measure of model performance. ML models can make more accurate classifications and predictions than their statistical counterparts and confer greater flexibility when modelling unstructured data or interactions between covariates; however, many ML models require larger sample sizes to achieve good classification or predictive performance and have been criticized as "black box" for their poor transparency and interpretability. ML holds potential in family medicine for risk profiling of patients' disease risk and clinical decision support to present additional information at times of uncertainty or high demand. In the future, ML approaches are positioned to become commonplace in family medicine. As such, it is important to understand the objectives that can be addressed using ML approaches and the associated techniques and limitations. This article provides a brief introduction into the use of ML approaches for classification and prediction tasks in family medicine.

The surface electromyography (sEMG) technique is proposed for muscle activation detection and intuitive control of prostheses or robot arms. Motion recognition is widely used to map sEMG signals to the target motions. One of the main factors preventing the implementation of this kind of method for real-time applications is the unsatisfactory motion recognition rate and time consumption. The purpose of this paper is to compare eight combinations of four feature extraction methods (Root Mean Square (RMS), Detrended Fluctuation Analysis (DFA), Weight Peaks (WP), and Muscular Model (MM)) and two classifiers (Neural Networks (NN) and Support Vector Machine (SVM)), for the task of mapping sEMG signals to eight upper-limb motions, to find out the relation between these methods and propose a proper combination to solve this issue. Seven subjects participated in the experiment and six muscles of the upper-limb were selected to record sEMG signals. The experimental results showed that NN classifier obtained the highest recognition accuracy rate (88.7%) during the training process while SVM performed better in real-time experiments (85.9%). For time consumption, SVM took less time than NN during the training process but needed more time for real-time computation. Among the four feature extraction methods, WP had the highest recognition rate for the training process (97.7%) while MM performed the best during real-time tests (94.3%). The combination of MM and NN is recommended for strict real-time applications while a combination of MM and SVM will be more suitable when time consumption is not a key requirement.

To advance time series forecasting we need to progress on multiple fronts. In this thesis, we develop algorithms to identify causal relations which allow to identify the driving processes containing useful information for the prediction of the process of interest. Complementing this, machine learning algorithms allow to exploit such information to build data-driven forecast models, and to correct the prediction of dynamical models. The identification from time series analysis of reliable indicators of causal relationships, is essential for many disciplines. Main challenges are distinguishing correlation from causality and discriminating between direct and indirect interactions. Over the years, many methods for data-driven causal inference have been proposed; however, their success largely depends on the characteristics of the system under investigation. Often, their data requirements, computational cost or number of parameters, limit their applicability. In this thesis, we propose a computationally efficient measure for causality testing, with the goal of overcoming the limitations of information-theoretic measures, due their high computational cost. The proposed metric is useful when causality networks need to be inferred from the analysis of a large number of relatively short time series. It can also be very useful for the selection of the inputs for the machine learning algorithms; in fact, it allows to identify those processes which contain useful information for the prediction of a given process. This is particularly useful for systems composed of a large number of processes, whose interactions are poorly understood. On the other hand, the socioeconomic impact of weather extremes draws the attention of researchers to the development of novel methodologies to make more accurate weather predictions. The Madden-Julian Oscillation (MJO), which is the dominant mode of variability in the tropical atmosphere on sub-seasonal time scales, is particularly important because it can promote or enhance extreme events in both, the tropics and the extratropics. Currently, the prediction skill of MJO is receiving a lot of attention, and in this thesis we take two machine learning approaches; first we use machine learning as a stand-alone technique to analyze observed data, showing that two artificial neural networks, a feed-forward neural network and a recurrent neural network, allow a competitive prediction, yet not exceeding the skill of the state-of-art dynamical models. Then, we combine dynamical models with machine learning, which allows to improve the predictions of the best dynamical model. In particular, machine learning allows to improve the prediction of the MJO intensity and geographical localization Para avanzar en el pronóstico de series temporales, es necesario avanzar en múltiples frentes. En esta tesis, desarrollamos algoritmos para descubrir relaciones causales que identifican los procesos que actúan como fuentes de información y pueden ayudar a mejorar la predicción del proceso de interés. Por otro lado, los algoritmos de aprendizaje automático permiten explotar dicha información para mejorar la predicción de los modelos dinámicos. La identificación de relaciones de causalidad a partir de series temporales es esencial en muchas disciplinas. Los desafíos en este ámbito son distinguir la correlación de la causalidad, así como diferenciar entre las interacciones directas e indirectas. A lo largo de los años se han propuesto numerosos métodos de inferencia causal basados en la observación de datos. No obstante, su éxito depende de las características del sistema a investigar. A menudo, el coste computacional o el número de parámetros limitan su aplicabilidad. En esta tesis se propone una medida computacionalmente eficiente para el testeo de causalidad. La métrica que se propone resulta util cuando es necesario inferir causalidad a partir de análisis de un gran número de series temporales relativamente cortas. También puede resultar muy útil en la selección de entradas en los algoritmos de aprendizaje automático. De hecho, permite identificar aquellos procesos que contienen información útil en la predicción de cierto proceso dado. Por otro lado, el impacto socioeconómico de fenómenos meteorológicos extremos requiere el desarrollo de nuevas metodologías con el objetivo de obtener predicciones meteorológicas más precisas. La Oscilación de Madden-Julian (MJO) es el modo dominante de variabilidad en la atmósfera tropical en escalas temporales subestacionales, y puede promover o aumentar eventos extremos tanto en el trópico como el extratrópico. Actualmente, la prediccion de la MJO genera mucho interés. Por esta razon, en esta tesis se han escogido dos metodologías diferentes de aprendizaje automático. Primero, se han utilizado dos redes neuronales artificiales para analizar datos observacionales, una red neuronal feed-forward y una red neuronal recurrente. Se muestra que esto permite una predicción competitiva, pero sin superar la capacidad de los modelos dinámicos actuales. Por este motivo, en un segundo estudio se han combinado modelos dinámicos con aprendizaje automático, que permiten mejorar las predicciones del mejor modelo dinámico. En particular, el aprendizaje automático permite mejorar la predicción de la intensidad de MJO y su localización geográfica

Summary The unsteady recovery of oil and gas prices in early 2017 led to an increase in drilling and hydraulic–fracturing operations in liquid–rich shale plays in North America. As field–development strategies continue to evolve, refracturing and infill–well drilling must be carefully combined to optimize shale–project profitability. Moreover, operators must bear in mind the undulating natural–gas demands persisting in an oversupplied shale–gas environment. In this paper, we use data–driven approaches to predict successful refracturing candidates and local gas demand for the second–tier optimization of a shale–gas supply–chain network. A strategic–planning (SP) model is developed for optimizing the net present value (NPV) of a case–study shale–gas network in the Marcellus Play. This SP model uses a mixed–integer–nonlinear–programming (MINLP) formulation developed in the General Algebraic Modeling System (GAMS, Release 27.1.0.2019). This model relies directly on input from reservoir simulation, local–gas–demand forecast, water–availability forecast, and natural–gas and West Texas Intermediate (WTI) crude–oil price forecasts. Before reservoir simulation, machine learning (ML) is used to predict successful refracturing candidates, using a feed–forward neural network (NN), random–forest (RF) classifier, and a t–distributed stochastic–neighbor–embedding (t–SNE) visualization technique. Using the obtained results, best–practice field–development strategies are implemented in the area of interest (AOI) using reservoir simulation. Local gas demand is forecasted using a long–short–term–memory (LSTM) recurrent NN (RNN) that uses a multivariate data set created from local and global variables affecting shale–gas demand. A water–management structure is also developed for the optimization framework. Using a 300–well data set (with 17 input features), successful refracturing candidates were proposed according to the joint outcome of an optimal 17/23/128/2 feed–forward NN, a t–SNE plot, and a techno–economic review. After ranking F1 scores, the developed NN outperforms the RF and support–vector–machine (SVM) algorithms for frac/refrac–well classification. The developed 32/256/128/120 LSTM model showed at least a 93% (±1%) prediction performance using three or five input features. The results illustrate the ability of the developed LSTM model to accurately predict local gas demands during periods of high or low gas demand. After SP optimization over a 10–year planning horizon, the economic results indicate an NPV of USD 481.945 million, using the proposed physics–data–driven–based approach. An NPV of USD 611.22 million is obtained when no ML was used. The results reveal that the application of ML to strategic planning can prevent erroneous feedback of project profitability while allowing early–time decision making that maximizes shale–asset NPV.

Identification of natural fractures in shale gas reservoirs using fracture signature function and machine learning models