Dendritic synaptic integration in central neurons

Prediction error, both at the level of sentence meaning and at the level of the next presented word, has been shown to successfully account for N400 amplitudes. Here we address the question of whether people differ in the representational level at which they implicitly predict upcoming language. To this end, we compute a measure of prediction error at the level of sentence meaning (magnitude of change in hidden layer activation, termed semantic update, in a neural network model of sentence comprehension, the Sentence Gestalt model) and a measure of prediction error at the level of the next presented word (surprisal from a next word prediction language model). When using both measures to predict N400 amplitudes during the reading of naturalistic texts, results showed that both measures significantly accounted for N400 amplitudes even when the other measure was controlled for. Most important for current purposes, both effects were significantly negatively correlated such that people with a reversed or weak surprisal effect showed the strongest influence of semantic update on N400 amplitudes, and random-effects model comparison showed that individuals differ in whether their N400 amplitudes are driven by semantic update only, by surprisal only, or by both, and that the most common model in the population was either semantic update or the combined model but clearly not the pure surprisal model. The current approach of combining large-scale models implementing different theoretical accounts with advanced model comparison techniques enables fine-grained investigations into the computational processes underlying N400 amplitudes, including interindividual differences.

The anti-interference characteristics of the neural network have a key impact on its information processing ability in complex environments. Most of the existing research focuses on small-scale networks and simplified models, and there is still a lack of systematic discussion on the influence mechanism of large-scale network expansion and topological complexity. In this study, a large-scale neural network model with different topologies is constructed to explore the influence mechanism of network size and connection complexity on the anti-disturbance characteristics. The optimal synchronization characteristics of complex NW small-world networks under noise interference are revealed, which provides a theoretical reference for the topology design and anti-interference ability improvement of artificial neural networks. Based on the Hodgkin-Huxley neuron dynamics model and Leonid chemical synapse theory, a complex Newman-Watts (NW) small-world network model containing 500 neurons is established for the first time, and the dynamic response characteristics of three topologies of simple ring network, simple NW small-world and complex NW small-world under music noise interference are compared and analyzed. The signal synchronization of the network is quantitatively evaluated by Pearson correlation, and the variation law of the anti-interference performance of different topologies is systematically revealed when the scale of the neural network is expanded from 100 to 500 neurons. The research shows that the expansion of network size and the increase of topological connection complexity can significantly enhance the anti-interference performance of neural network. Among them, the complex NW small-world network performs best in the noise interference environment, and the correlation coefficient increases significantly at the scale of 500 neurons. In this study, the network scale is extended to 500 neurons for the first time. By constructing a complex NW small-world topology, the influence of scale expansion and connection complexity improvement on the network anti-interference performance is systematically quantified, which provides reference simulation data for the simulation research of artificial neural networks.

At present, the accuracy of text detection from natural scene images based on deep learning is much higher than that of traditional text detection algorithm. However, most of the text detectors based on deep learning should use large-scale neural network models (such as VGG and ResNet) for improving the accuracy. Because the weight files of these models are very large (VGG is about 500MB and ResNet is about 100MB), they are not suitable for the computationally limited platforms in daily life. Based on the small-scale neural network model named MobileNet V2, and U-Net that is used in semantic segmentation commonly, a text detector for natural scene by using neural network with low complexity is proposed in this paper. The weight file of the designed neural network model is about 16MB, which is suitable for running on mobile platforms. The experiment result based on the ICDAR 2013 dataset proves that this model has good performance in text detection.

In order to address the data security and communication efficiency of vehicles during high-speed mobile communication, this paper investigates the problem of secure in-vehicle communication resource allocation based on slow-variable large-scale fading channel information, to meet the quality of service requirements of vehicular communication, i.e., to ensure the reliability of V2V communication and the time delay while maximizing the transmission rate of the cellular link. And an eavesdropping model is introduced to ensure the secure delivery of link information. Considering that the high mobility of vehicles causes rapid channel changes, we model the problem as a Markov decision process and propose a resource allocation optimization framework based on the Multi-Agent Reinforcement Learning Algorithm (MARL-DDQN), in which a large-scale neural network model is built to train vehicular to learn the optimal resource allocation strategy for optimal communication performance and security performance. Simulation results show that the load successful delivery rate and confidentiality performance of the vehicular communication network are effectively improved compared to the baseline and MADDPG strategies while ensuring link security. This study provides useful references and practical value for the optimization of secure communication resource allocation in vehicular networking.

The past few years have witnessed the flourishing of large-scale deep neural network models with ever-growing parameter numbers. Training such large-scale models typically requires massive memory and computing resources, necessitating distributed training. As GPU performance has rapidly evolved in recent years, computation time has shrunk, making communication a larger portion of the overall training time. Consequently, optimizing communication for distributed training has become crucial. In this article, we briefly introduce the general architecture of distributed deep neural network training and analyze relationships among Parallelization Strategy, Collective Communication Library, and Network from the perspective of communication optimization, which forms a three-layer paradigm. We then review current representative research advances within this three-layer paradigm. We find that layers in the current three-layer paradigm are relatively independent and there is a rich design space for cross-layer collaborative optimization in distributed training scenarios. Therefore, we advocate “Vertical” and “Horizontal” co-designs which extend the three-layer paradigm to a five-layer paradigm. We also advocate “Intra-Inter” and “Host-Net” co-designs to further utilize the potential of heterogeneous resources. We hope this article can shed some light on future research on communication optimization for distributed training.

Network models incorporating chloride dynamics predict optimal strategies for terminating status epilepticus.

Progress towards the energy breakthroughs needed to combat climate change can be significantly accelerated through the efficient simulation of atomistic systems. However, simulation techniques based on first principles, such as density functional theory (DFT), are limited in their practical use due to their high computational expense. Machine learning approaches have the potential to approximate DFT in a computationally efficient manner, which could dramatically increase the impact of computational simulations on real-world problems. However, they are limited by their accuracy and the cost of generating labeled data. Here, we present an online active learning framework for accelerating the simulation of atomic systems efficiently and accurately by incorporating prior physical information learned by large-scale pre-trained graph neural network models from the Open Catalyst Project. Accelerating these simulations enables useful data to be generated more cheaply, allowing better models to be trained and more atomistic systems to be screened. We also present a method of comparing local optimization techniques on the basis of both their speed and accuracy. Experiments on 30 benchmark adsorbate-catalyst systems show that our method of transfer learning to incorporate prior information from pre-trained models accelerates simulations by reducing the number of DFT calculations by 91%, while meeting an accuracy threshold of 0.02 eV 93% of the time. Finally, we demonstrate a technique for leveraging the interactive functionality built in to Vienna ab initio Simulation Package (VASP) to efficiently compute single point calculations within our online active learning framework without the significant startup costs. This allows VASP to work in tandem with our framework while requiring 75% fewer self-consistent cycles than conventional single point calculations. The online active learning implementation, and examples using the VASP interactive code, are available in the open source FINETUNA package on Github.

Impacts of vehicle electrification on large-scale transportation and charging infrastructure: A dynamic network modeling approach

Spatial information is encoded by the hippocampus, and the factors that contribute to the amount of information that can be encoded and the transformation of spatial information through the trisynaptic circuit remain an important issue. A large-scale neuronal network model of the rat entorhinal-dentate system was developed with multicompartmental representations of the neurons within the dentate gyrus. Spatial information was introduced to the network via grid cell activity, and the spatial information encoding capabilities of the network were assessed using a recursive decoding algorithm to estimate the position of a virtual rat using the dentate activity. To obtain a measure for the information that the network could convey, decoding error was calculated for different decoding population sizes. Decoding error decreased exponentially as a function of population size. Therefore, the time constant and the asymptote of the error curve could be used as metrics to compare the changes in encoding performance. In conjunction with the large-scale model, this paradigm can be used to characterize how neural properties, network composition, and the interactions between different subfields affect spatial information encoding.

The scale of model parameters and datasets is rapidly growing for high accuracy in various areas. To train a large-scale deep neural network (DNN) model, a huge amount of computation and memory is required; therefore, a parallelization technique for training large-scale DNN models has attracted attention. A number of approaches have been proposed to parallelize large-scale DNN models, but these schemes lack scalability because of their long communication time and limited worker memory. They often sacrifice accuracy to reduce communication time. In this work, we proposed an efficient parallelism strategy named group hybrid parallelism (GHP) to minimize the training time without any accuracy loss. Two key ideas inspired our approach. First, grouping workers and training them by groups reduces unnecessary communication overhead among workers. It saves a huge amount of network resources in the course of training large-scale networks. Second, mixing data and model parallelism can reduce communication time and mitigate the worker memory issue. Data and model paralleism are complementary to each other so the training time can be enhanced when they are combined. We analyzed the training time model of the data and model parallelism, and based on the training time model, we demonstrated the heuristics that determine the parallelization strategy for minimizing training time. We evaluated group hybrid parallelism in comparison with existing parallelism schemes, and our experimental results show that group hybrid parallelism outperforms them.

Memristors have emerged as promising devices for enabling efficient multiply-accumulate (MAC) operations in crossbar arrays, crucial for analog in-memory computing (AiMC). However, variations in memristors and associated circuits can affect the accuracy of analog computing. Typically, this is mitigated by on-chip training, which is challenging for memristors with limited endurance. We present a hardware-software codesign using magnetic tunnel junction (MTJ)-based AiMC off-chip calibration that achieves software accuracy without costly on-chip training. Hardware-wise, MTJ devices exhibit ultralow cycle-to-cycle variations, as experimentally evaluated over 1 million mass-produced devices. Software-wise, leveraging this, we propose an off-chip training method to adjust deep neural network parameters, achieving accurate AiMC inference. We validate this approach with MAC operations, showing improved transfer curve linearity and reduced errors. By emulating large-scale neural network models, our codesigned MTJ-based AiMC closely matches software baseline accuracy and outperforms existing off-chip training methods, highlighting MTJ's potential in AI tasks.

Training large-scale neural network models is computationally expensive and demands a great deal of resources. It is an important area of study with a lot of potential for the future of the AI industry. In recent years, the power of computer hardware has significantly improved and we have new breakthroughs in deep learning. With these innovations, the computational cost of training large neural network models has declined by at least 10 folds in high- and average-performance machines. In this research, we explore NAL, AutoML, and other frameworks to determine the best suitable model for edge devices. The biggest improvements compared to reference models can be acquired if the NAS algorithm is co-designed with the corresponding inference engine.

In this study, we investigated one type of auditory perceptual grouping phenomena--the auditory continuity illusion (also called temporal induction). We employed a previously developed, neurobiologically realistic, large-scale neural network model of the auditory processing pathway in the cortex, ranging from the primary auditory cortex to the prefrontal cortex, and simulated temporal induction without changing any model parameters. The model processes tonal contour stimuli, composed of combinations of upward and downward FM sweeps and tones, in a delayed match-to-sample task. The local electrical activities of the neuronal units of the model simulated accurately the experimentally observed electrophysiological data, where available, and the model's simulated BOLD-fMRI data were quantitatively matched with experimental fMRI data. In the present simulations, intact stimuli were matched with fragmented versions (i.e., with inserted silent gaps). The ability of the model to match fragmented stimuli declined as the duration of the gaps increased. However, when simulated broadband noise was inserted into these gaps, the matching response was restored, indicating that a continuous stimulus was perceived. The electrical activities of the neuronal units of the model agreed with electrophysiological data, and the behavioral activity of the model matched human behavioral data. In the model, the predominant mechanism implementing temporal induction is the divergence of the feedforward connections along the auditory processing pathway in the temporal cortex. These simulation results not only attest to the robustness of the model, but further predict the primary role of the anatomical connectivity of the auditory processing areas in mediating the continuity illusion.

Realtime cerebellum: A large-scale spiking network model of the cerebellum that runs in realtime using a graphics processing unit

We propose a large-scale sparse customer-facility network model that allows a customer to be assigned only to facilities within the vicinity of a customer. In this model, customer-facility distances are integer values representing zones. Experimental results are presented for large instances with up to 100,000 customers and 100 potential facility sites. A mixed-integer linear programming solver reveals large gaps in suboptimal solutions and lower bounds provided, even with a considerable computational effort. Two simple but scalable local search heuristics are computationally investigated, revealing their potential for solving such large-scale problems in practice.