Использование методов машинного обучения для решения задачи энергооптимального движения поезда

Constructive training of recurrent neural networks using hybrid optimization

Training of neural network can be accelerated by limited numerical precision together with specialized low-precision hardware. This paper studies how low precision can impact on entire training of RNNs. We emulate low precision training for recently proposed gated recurrent unit (GRU) and use dynamic fixed point as a target numeric format. We first show that batch normalization on input sequences can help speed up training with low precision as well as high precision. We also show that the overflow rate should be carefully controlled for dynamic fixed point. We study low precision training with various rounding options including bit truncation, round to nearest, and stochastic rounding. Stochastic rounding shows superior results than the other options. The effect of fully low precision training is also analyzed by comparing partial low precision training. We show that the piecewise linear activation function with stochastic rounding can achieve comparable training results with floating point precision. Low precision multiplier and accumulator (MAC) with linear-feedback shift register (LFSR) is implemented with 28nm Synopsys PDK for energy and performance analysis. Implementation results show low precision hardware is 4.7x faster, and energy per task is up to 4.55x lower than that of floating point hardware.

In this paper, we use block-diagonal matrix of approximate the Hessian matrix in the Levenberg Marquardt method during the training of recurrent neural networks. We analyze the weight updating strategies and the groupings of the weights associated with the approximation. Two weight updating strategies, namely asynchronous and synchronous updating methods are investigated. Asynchronous method updated weights of one block at a time while synchronous method updates all weights at the same time. Variations of these two methods, which involve the determination of two parameters mu and lambda, are examined. Four weight grouping methods, correlation blocks, k-unit blocks, layer blocks and arbitrary blocks are investigated and compared. Their computational complexity, approximation ability, and training time is analyzed. Comparing with the original Levenberg Marquardt method, the block-diagonal approximation methods give substantial improvement in training time without degrading the generalization ability.

This article is devoted to the creation of a model of observational diagnostics for patients in dentistry using artificial intelligence as an information and measurement system. Artificial intelligence technology and algorithms in the field of clinical and radiographic image analysis in general dentistry and orthopedics have become the most common systems, contributing to accurate and timely diagnosis of a particular patient. Along with clinical practice images, radiographs are key tools for graphic visualization of the patient's dentition. A model of observational diagnostics for patients using artificial intelligence technology as the latest tool in the healthcare industry can become a structured method of information standardization and measurements in general dentistry. The model of observational diagnostics can serve as a fundamental component of functional diagnostics based on artificial intelligence algorithms, since they have a special potential for increasing the reliability and consistency of diagnostic measurements and information used by a dentist during treatment planning and restoration of the dentition. The model of observational diagnostics of dental patients presented by the authors implies a significant step forward in the development of highprecision methods of functional diagnostics, monitoring and analysis of patient condition measurements. The algorithm of the observational diagnostics model was integrated into the program, the operation of which is based on artificial intelligence technology in order to increase the efficiency, consistency, optimality and accuracy of the processes carried out in clinical practice. The purpose of the study is to determine the characteristics of the model of observational diagnostics of dental patients using artificial intelligence as an information and measuring system. Main results of the study: 1) the characteristics of the practical application of artificial intelligence technology for radiographs, clinical images of patients' teeth are provided; 2) the essential aspects of the model of observational diagnostics of dental patients, as well as the features of its practical application for planning dental treatment of patients are highlighted; 3) the results of practical testing of the model of observational diagnostics of dental patients are reflected (using the example of detection and treatment of caries). Research materials: the results of academic studies on the assessment of functional diagnostics of oral cavity and dentition pathologies using artificial intelligence technology; practical testing of the model of observational diagnostics for dental patients (using the example of detection and treatment of caries in a 49yearold woman). Research methods: systematization, comparative analysis, contrastive analysis, generalization, abstractlogical method, induction, deduction, systems approach, data conceptualization, graphical and tabular visualization of data, construction of informationmeasuring systems, artificial intelligence methods, study of clinical images and radiographs of patients' dental arches. The main conclusions of the study: 1) information and measuring instruments, the operation of which is based on artificial intelligence technology, have become the basis for the rapid processing of significant amounts of information, increasing the accuracy and efficiency of clinical decisions in dentistry. These information and measuring systems allow us to determine complex patterns on radiographs and clinical images of dental arches, to increase the degree of accuracy of functional diagnostics, in particular, in the presence of subtle pathologies and disorders; 2) artificial intelligence technology cannot replace a qualified dentist, such information and measuring systems only complement the accumulated human professional experience, allowing us to jointly carry out functional diagnostics of patients with a dental profile; 3) the potential of artificial intelligence technologies to create comprehensive plans and reports for dental treatment and dental restoration forms the basis for the joint activities of the latest tools and a dentist. In the context of the study, it is noted that artificial intelligence algorithms have become a powerful informationmeasuring tool for improving the decisionmaking process in dental practice, since digital technology promptly and quickly provides feedback to the dentist about missed anomalies or suboptimal methods of dental treatment and restorative medicine. Integration of the model of observational diagnostics for dental patients will make it possible to form an algorithm for the operation of artificial intelligence that fully complies with the recommendations for functional diagnostics and dental treatment of patients, developed on the basis of factual information. The use of artificial intelligence algorithms in dental practice will increase the efficiency of the general industry standard for providing dental services to patients.

BACKGROUND: Currently, artificial intelligence in the form of artificial neural networks is being actively implemented in a number of areas of our lives, including medicine. In particular, in otorhinolaryngology, artificial neural networks are used to analyze images obtained during endoscopic examinations of patients (e.g., videolaryngoscopy) [1–3]. The interpretation of laryngoscopic images often presents significant difficulties for practicing physicians, which reduces the frequency of detection of precancerous laryngeal diseases and contributes to the increase in the number of patients with stage III–IV laryngeal cancer [4, 5]. This underscores the significance of prompt performance and accurate interpretation of the findings of endoscopic examinations of patients with laryngeal disorders. Artificial neural networks can be employed to analyze the results of videolaryngoscopy, furnishing the physician with supplementary information that can enhance diagnostic accuracy and diminish the probability of error [6, 7]. AIM: The study aims to develop and train an artificial neural network for recognizing characteristic features of laryngeal neoplasms and variants of laryngeal normality. MATERIALS AND METHODS: The study was conducted under the grant of the Moscow Center for Innovative Technologies in Healthcare (grant No. 2112-1/22) entitled “Using Neural Networks (Artificial Intelligence Algorithms) for Control and Improving the Quality of Diagnosis and Treatment of Diseases of Laryngeal and Ear Structures through Digital Technologies”.The following methods were used during the course of the study: data collection for the creation of a photobank (dataset) of medical images obtained during videolaryngoscopy; data partitioning for the formation of datasets for individual nosologies and groups of diseases; the method of consilium; analysis of the accuracy of recognition and classification of digital endoscopic images; and training of classification neural networks. Consequently, a dataset comprising 1,471 laryngeal images in digital formats (JPEG, BMP) was assembled, labelled, and uploaded for the purpose of training the artificial neural network. Of the total number of images, 410 were classified as pertaining to laryngeal formation, while 1061 were classified as variants of normality. Subsequently, the neural network was trained and tested to identify the signs of normal and laryngeal masses. RESULTS: The results of the testing of the artificial neural network indicated the formation of an inaccuracy matrix, the calculation of the value of recognition accuracy, the calculation of the quality indicators of the model performance, and the construction of the ROC curve. The developed and trained artificial neural network demonstrated an accuracy of 86% in recognizing the signs of laryngeal masses and norms. CONCLUSIONS: This study demonstrates that a trained artificial neural network can successfully distinguish between signs of normal and laryngeal masses in endoscopic photographs. With further training of the neural network and achievement of high accuracy, this technology can be used in clinical practice as an assistant in the interpretation of laryngoscopic images and early diagnosis of laryngeal masses. It can also be employed to control and improve the quality of diagnosis and treatment of diseases of the throat, nose, and ears by primary care physicians.

not provided.

Methods such as Layer Normalization (LN) and Batch Normalization have proven to be effective in improving the training of Recurrent Neural Networks (RNNs). However, existing methods normalize using only the instantaneous information at one particular time step, and the result of the normalization is a preactivation state with a time-independent distribution. This implementation fails to account for certain temporal differences inherent in the inputs and the architecture of RNNs. Since these networks share weights across time steps, it may also be desirable to account for the connections between time steps in the normalization scheme. In this paper, we propose a normalization method called Assorted-Time Normalization (ATN), which preserves information from multiple consecutive time steps and normalizes using them. This setup allows us to introduce longer time dependencies into the traditional normalization methods without introducing any new trainable parameters. We present theoretical derivations for the gradient propagation and prove the weight scaling invariance property. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks, such as Adding, Copying, and Denoise Problems and Language Modeling Problems.

Unintentional falls can cause severe injuries and even death, especially if no immediate assistance is given. A fall detection system aims to detect a fall as soon as it occurs, therefore issuing an automatic assistance request. Wearable embedded sensors are emerging as the most viable solution for continuous monitoring since they are more effective, less intrusive and less expensive than other systems. Tailoring a deep learning method to the requirements of microcontrollers entails matching very stringent constraints in terms of both memory and computational power. In addition, datasets acquired with wearable devices are relatively scarce and not necessarily devised for supervised learning. In this work, we discuss the design of a software architecture based on recurrent neural networks which can be effective for fall detection while running entirely onboard a wearable device. The well-known and publicly-available SisFall dataset was adopted and extended with fine-grained temporal annotations to cope with the supervised training of recurrent neural networks. We then show that an appropriate process of architectural minimization together with accurate hyperparameters selection leads to a workable model which compares favorably with other detection techniques. The embedding of the resulting architecture has been validated using a state-of-art hardware device.

We investigate how unsupervised training of recurrent neural networks (RNNs) and their deep hierarchies can benefit a supervised task like temporal pattern detection. The RNNs are fully and fast trained by unsupervised algorithms and only supervised feed-forward readouts are used. The unsupervised RNNs are shown to perform better in a rigorous comparison against state-of-art random reservoir networks. Unsupervised greedy bottom-up trained hierarchies of such RNNs are shown being capable of big performance improvements over single layer setups.

The main task of using neural networks is the prompt and accurate solution of various creative tasks, including the analysis and synthesis of news flows, while maintaining the continuity of learning. The result of such processing can be digests, filtered news streams, as well as event forecasts that allow for proactivity in management decisions. Known methods of news processing by neural networks and technical solutions that implement them do not fully provide a solution to the problems that arise in this area. It is necessary to expand their functionality, and improve the space-time signal binding in recurrent neural networks. When processing news flows, simultaneously with continuous training of recurrent neural networks, selection, recognition, restoration, prediction and synthesis of news should be carried out. To reduce the severity of the problem, a promising method of multifunctional processing of news flows is proposed using recurrent neural networks with a logical organization of layers and continuous learning. The method is based on the development of associative processing of textual information in streaming recurrent neural networks with controlled elements. The key features of this method are the multifunctional processing of information flows with changing laws of news appearance. The method provides for operational selection, recognition, restoration, forecasting and synthesis of news based on deep associative continuous processing of links between text elements. The neural network system that implements the proposed method differs from the known solutions by new elements, connections between them, as well as by the functions performed. The results of the experiments confirmed the extended functionality of the method. New features of processing news texts by streaming RNNs are revealed. The proposed solutions can be used to create a new generation of intelligent systems not only for word processing, but also for other types of information.

Reservoir Computing (RC) is a paradigm for efficient training of Recurrent Neural Networks (RNNs). The Echo State Network (ESN), a type of RC paradigm, has been widely used for time series forecasting. Whereas, few works exist on classification with ESN. In this paper, we shed light on the use of ESN for pattern recognition problem, i.e. emotion recognition from Electroencephalogram (EEG). We show that the reservoir with its recurrence is able to perform the feature extraction step directly from the EEG raw. Such kind of recurrence rich of nonlinearities allows the projection of the input data into a high dimensional state space. It is well known that the ESN fails due to the poor choices of its initialization. Nevertheless, we show that pretraining the ESN with the Intrinsic Plasticity (IP) rule remedies the shortcoming of randomly initialization. To validate our approach, we tested our system on the benchmark DEAP containing EEG signals of 32 subjects and the results were promising.

[This corrects the article DOI: 10.3389/fnins.2024.1439155.].

We present RNNbow, an interactive tool for visualizing the gradient flow during backpropagation in training of recurrent neural networks. By visualizing the gradient, as opposed to activations, RNNbow offers insight into how the network is learning. We show how it illustrates the vanishing gradient and the training process.

In this paper we demonstrate a modified topology for long short-term memory recurrent neural networks that controls the shape of the squashing functions in gating units. We further propose an efficient training framework based on a mini-batch training on sequence level combined with a sequence chunking approach. The framework is evaluated on publicly available data sets containing English and French handwriting by utilizing a GPU based implementation. Speedups of more than 3x are achieved in training recurrent neural network models which outperform state of the art recognition results.

Vandermonde-subspace frequency division multiplexing (VFDM) utilizes frequency-domain degrees of freedom (DoFs) to implement spectrum sharing. In this paper, a robust intra-tier precoder (ITP) is devised to suppress the interference for VFDM based small cell (SC) considering imperfect Gaussian distributed estimation error. We formulate the ITP design as a minimum mean square error (MMSE) optimization, where the total mean square error (MSE) is carefully derived. Considering the complexity of the solution, we then proposed an iterative algorithm to find the optimized robust ITP. Numerical results are presented to validate that the proposed algorithm can achieve lower mean square error (MSE) than the traditional one does.