An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning

Modulation format identification (MFI) and optical signal-to-noise ratio (OSNR) monitoring are essential for elastic optical networks. A method for joint modulation format identification and OSNR monitoring based on ternary neural networks was proposed. Further, a ternary neural network was established, and the constellation images after constant modulus algorithm (CMA) equalization were used as input features. Four commonly used modulation formats were distinguished, including dual-polarization (DP)-QPSK, 8QAM, 16QAM, and 64QAM. A 32G baud simulation system was constructed, and the results were analyzed. First, we investigated the influence of resolution and sample length on the identification accuracy. The values 32 and 7000 were selected for the two parameters, respectively, based on the balance between resource consumption and accuracy. Then, we compared the performance of the binary neural network (BNN), ternary neural network (TNN), and full-precision neural network (FNN). The results indicate that the memory consumption and extraction time of the TNN are similar to those of the BNN, and the accuracy is further improved. Moreover, the robustness of the system was analyzed. The results validate that the proposed method can tolerate fiber nonlinearity (from 500 km to 1500 km). Finally, an experiment was conducted to prove the practicality of the proposed method.

This work presents an FPGA-based hardware/software design to help the agricultural robot intelligently decide if biological agents need to be applied to the target crops. For target crop recognition, in our global positioning, the selective search integrates with a thresholding scheme to reduce the number of region of interest (ROI) in a captured image. In our local recognition, a binarized neural network (BNN) architecture is presented to help recognize the target crop. Furthermore, an estimation method of pest and disease severity is also presented. Experiments show integrating our presented BNN architecture needs a few extra resources (less than 17% of available FPGA resources in terms of Xilinx Zynq UltraScall+™ MPSoC ZU3EG A484), compared to an existing BNN one. However, the top-1 accuracy rate and the top-5 one can be increased by 32.25% to 32.84% and by 14.99% to 15.17%, respectively. Furthermore, when the presented BNN architecture was also implemented on the ARM Cortex-A53 CPU and the NVIDIA GeForce RTX 2080 GPU, our BNN hardware module on the FPGA can accelerate the frames per second (FPS) by a factor of 3,690.18 and a factor of 1.07, respectively.

Traffic flow optimization in urban environments remains one of the key challenges of modern research, even despite the significant volume of scientific works devoted to this topic. Despite the achievements, this problem still does not have a universal solution that would work effectively in real-world scenarios. One of the main difficulties is the processing of a large array of input data, in particular, traffic data, which constantly comes from sensors installed throughout the urban road network. Traditionally, due to the scale of the task, researchers have focused on the development of systems with localized agents. Such agents usually manage traffic at individual intersections, while their coordination is carried out within the framework of multi-stream agent systems. However, modern approaches take into account the volume and complexity of input data through the use of deep learning methods. In particular, the use of the deep deterministic policy gradient (DDPG) algorithm is proposed, on the basis of which large input data can be processed. As part of the experimental study, a simple intersection model was tested to verify the effectiveness of the approach. The DDPG algorithm performed better in the simple model compared to Q-learning. DDPG provided a reward in the range of 4-4.3 points, while the reward of Q-learning was in the range of 2-4 points. To evaluate the performance of the DDPG approach compared to Q-learning and random timings, the main criterion is the average reward per episode. DDPG and Q-learning achieve similar reward levels, but DDPG shows stable convergence (0.04-0.21 points), while Q-learning remains unstable (0.04-0.43 points). The study of intra-episode performance shows that DDPG achieves improvements mainly closer to the end of the episode. Overall, this algorithm has proven successful for this scenario, and the results obtained can serve as a basis for further improvements and applications in more complex traffic scenarios.

The article focuses on policy learning processes with the aim of sharing knowledge about how policy learning happens. In doing this, it considers conflict as a natural process in policy learning and proposes action research as a possible strategy to address it. By reflecting on a long-term action research process in the province of Gipuzkoa (Basque Country, Spain), the article proposes a change of focus in assessing the role of context. If we assume that policy learning is oriented towards changing the context, then context plays a more complex and important role than it is usually given in theory. At the core of this argument is the understanding that the cognitive frameworks of policy makers and researchers are also part of context and, when changing the context, participants in policy learning are involved in a change process which alters the very same cognitive frameworks that conditioned the initiation of policy learning.

The convolutional neural network (CNN) is a significant part of the artificial intelligence (AI) systems widely used in different tasks. The binarized neural networks (BNNs) reduce power consumption and hardware overhead to answer the demands for using AI in power-limited applications. In this paper, a BNN hardware accelerator is proposed. The proposed approach is based on a novel nonvolatile XNOR/XOR circuit designed using the magnetic tunnel junction (MTJ) and gate-all-around carbon nanotube field-effect transistor (GAA-CNTFET) devices. The nonvolatility of the proposed design leads to the elimination of external memory access that significantly decreases the data transmission delay and power dissipation. Moreover, it consumes low energy, which is very critical in battery-operated devices. Furthermore, the combinational read circuitry of the proposed design leads to high robustness to process variations. According to the simulation results, our proposed design has a logical error rate of 0.0164%, which is negligible and offers a significantly high network accuracy even in the presence of significant process variations. Our proposed hardware accelerator provides at least 13%, 29%, and 41% improvements regarding power, power delay product (PDP), and area compared to its state-of-the-art counterparts.

RL-VAEGAN: Adversarial defense for reinforcement learning agents via style transfer

This paper investigates how to efficiently transition and update policies, trained initially with demonstrations, using off-policy actor-critic reinforcement learning. It is well-known that techniques based on Learning from Demonstrations, for example behavior cloning, can lead to proficient policies given limited data. However, it is currently unclear how to efficiently update that policy using reinforcement learning as these approaches are inherently optimizing different objective functions. Previous works have used loss functions, which combine behavior cloning losses with reinforcement learning losses to enable this update. However, the components of these loss functions are often set anecdotally, and their individual contributions are not well understood. In this work, we propose the Cycle-of-Learning (CoL) framework that uses an actor-critic architecture with a loss function that combines behavior cloning and 1-step Q-learning losses with an off-policy pre-training step from human demonstrations. This enables transition from behavior cloning to reinforcement learning without performance degradation and improves reinforcement learning in terms of overall performance and training time. Additionally, we carefully study the composition of these combined losses and their impact on overall policy learning. We show that our approach outperforms state-of-the-art techniques for combining behavior cloning and reinforcement learning for both dense and sparse reward scenarios. Our results also suggest that directly including the behavior cloning loss on demonstration data helps to ensure stable learning and ground future policy updates.

Learning policies that generalize across multiple tasks is an important and challenging research topic in reinforcement learning and robotics. Training individual policies for every single potential task is often impractical, especially for continuous task variations, requiring more principled approaches to share and transfer knowledge among similar tasks. We present a novel approach for learning a nonlinear feedback policy that generalizes across multiple tasks. The key idea is to define a parametrized policy as a function of both the state and the task, which allows learning a single policy that generalizes across multiple known and unknown tasks. Applications of our novel approach to reinforcement and imitation learning in real-robot experiments are shown.

Cities are expected to contribute to climate mitigation by providing effective climate policy actions in a multilevel governance context. Yet, it is unclear what we actually know about the strengths and weaknesses of urban climate policy measures and how municipal policy learning could be enhanced. In this article we present a comprehensive review on cities’ experience in climate mitigation policy and policy learning, focusing on research published in journal articles. We provide observations from the literature on municipal experience in policy practice and on how local governments manage learning in urban mitigation policy. The review provides valuable examples and reflections on individual policy initiatives, both in terms of policy framing and policy instruments, but finds sporadic evidence on systematically successful policy learning and capacity building processes in the scientific literature. The number of scientific articles on empirical assessments of local policy learning are few, and the research found, published in numerous journals, potentially addressing different research communities, provides limited knowledge on policy learning in terms of lessons learned, lessons drawing and capacity building. Moreover, the literature builds mostly on case studies within particular institutional contexts, which makes the transferability of findings problematic. While evidence on policy learning is still limited, there are indications that under the right conditions, cities can “learn to learn”.

Reinforcement Learning (RL) has shown great performance in solving sequential decision-making and control in dynamic environments problems. Despite its achievements, training Deep Neural Network (DNN) based RL is expensive in terms of time and power because of the large number of episodes required to train agents with high dimensional image representations. At the deployment also, the massive energy footprint of deep neural networks can be a major drawback. Embedded devices as the main deployment platform, are intrinsically resource-constrained and deploying DNN on them is challenging. Consequently, reducing the number of actions taken by the RL agent to learn desired policy, along with the development of efficient hardware architectures for RL is crucial. In this paper, we propose a novel hardware architecture for RL agents based on the learning hierarchical policies method. We show that hierarchical learning with several levels of control improves RL agents training efficiency and the agent converges faster compared to a none hierarchical model and therefore using less power. This is especially true as the environment becomes more complex with multiple objective sub-goals. Our method is important for efficient learning of policies for RL agent, especially when the target platform is a resource constraint embedded device. By performing a systematic neural network architecture search and hardware design space exploration, we implemented an energy-efficient scalable hardware accelerator for the hierarchical RL. Hardware factors of merit such as the latency, throughput, and energy consumption of the accelerator are evaluated with the various processing elements, and model parameters. The most energy-efficient configuration achieves 139 fps throughput with 5.8 mJ energy consumption per classification on Xilinx Artix-7 FPGA. Compared to similar works our design shows up to 3x better energy efficiency.

In the last few years, there is an increasing demand for developing efficient solutions for computer vision-related tasks on FPGA hardware due to its quick prototyping and computing capabilities. Therefore, this work aims to implement a low precision Binarized Neural Network (BNN) using a Python framework on the Xilinx PYNQZ2 embedded platform to tackle the challenging problem of Sign Language recognition. More specifically, the FINN framework is adopted and the BNN topology is modified to adapt large resolution (i.e 64x64) to perform classification of proposed Indian Sign Language (ISL) gestures into corresponding numbers. In addition, data augmentation techniques are also applied to improve the overall performance of the neural network. Furthermore, hardware/software co-verification of BNN topology is performed to validate the accuracy after implementing it onto hardware. Extensive experimental results show that it achieves a classification rate of 843.8 frames per second (FPS) on PYNQ-Z2 FPGA which delivers higher performance as compared to previous works. Also, the post-implementation results are analyzed in terms of resource utilization and power consumption.

The search for interpretable reinforcement learning policies is of high academic and industrial interest. Especially for industrial systems, domain experts are more likely to deploy autonomously learned controllers if they are understandable and convenient to evaluate. Basic algebraic equations are supposed to meet these requirements, as long as they are restricted to an adequate complexity. In our recent work "Interpretable policies for reinforcement learning by genetic programming" published in Engineering Applications of Artificial Intelligence 76 (2018), we introduced the genetic programming for reinforcement learning (GPRL) approach. GPRL uses model-based batch reinforcement learning and genetic programming and autonomously learns policy equations from preexisting default state-action trajectory samples. Experiments on three reinforcement learning benchmarks demonstrate that GPRL can produce human-interpretable policies of high control performance.

Interpretable policies for reinforcement learning by genetic programming

Policy actors involved in decision-making processes interact and gradually accumulate evidence about policy problems and solutions. As a result, they update their policy beliefs and preferences over time. This process of policy learning is consistent if policy preferences are aligned with any adaptations in beliefs about policy outcomes – a crucial condition of learning-induced policy changes. This article examines whether and when policy learning is consistent based on regression analyses conducted on data from a 2012 survey of 293 Belgian actors involved in the European liberalization policy process for the rail and electricity sectors. In line with the advocacy coalition framework, existing research has suggested that motivated modes of reasoning, such as selective exposure and biased assimilation, influence policy actors’ attitudes and behaviours. This study isolates the effect of biased assimilation on policy learning by demonstrating that when policy actors adapt their beliefs about policy outcomes, they do not necessarily align their policy preferences with those adaptations. Furthermore, biased assimilation is higher among politically curious actors, but their degree of commitment to the policy process does not appear to play a role. The theoretical and practical implications of these findings are discussed.

Despite the achievements of recent binarization methods on reducing the performance degradation of Binary Neural Networks (BNNs), gradient mismatching caused by the Straight-Through-Estimator (STE) still dominates quantized networks. This paper proposes a meta-based quantizer named QuantNet, which utilizes a differentiable sub-network to directly binarize the full-precision weights without resorting to STE and any learnable gradient estimators. Our method not only solves the problem of gradient mismatching, but also reduces the impact of discretization errors, caused by the binarizing operation in the deployment, on performance. Generally, the proposed algorithm is implemented within a fully differentiable framework, and is easily extended to the general network quantization with any bits. The quantitative experiments on CIFAR-100 and ImageNet demonstrate that QuantNet achieves the signifficant improvements comparing with previous binarization methods, and even bridges gaps of accuracies between binarized models and full-precision models.