A Reinforcement-Learning-Based Deployment Strategy for GPP Components

Efficient and effective broadband wireless technologies are required to support the foreseeable need to process and exchange large amounts of data. In this Special Issue of the Springer Journal of Signal Processing Systems various disciplines of such efficiency and effectiveness in the context of Software-Defined Radio (SDR) technologies are discussed. At first, a selection of articles focuses on “spectral efficiency”. Since frequency spectrum is known to be scarce and limited, it becomes a resource of increasing value whose utilization must be handled with care. The rapidly growing demand for higher data throughputs in wireless communication does not allow inefficient use of spectrum anymore. Two promising approaches to tackle the “spectral efficiency” issue are Cognitive Radio (CR) and Dynamic Spectrum Access (DSA). A Cognitive Radio is capable of learning and adapting to its transmission environment. Taking e.g. spectrum sensing results into account, the decisions for accessing the scarce frequency spectrum in the most flexible and effective manner are made. CRs are typically based on SDR technology. Thus, the first set of articles puts emphasis on topics in the areas of Cognitive Radio, Cognitive Radio Networks, and Spectrum Sensing. The second selection of articles addresses the problems of “energy efficiency” and “implementation efficiency”. Especially in wireless communications these types of efficiencies are highly important to increase battery time of modern SDRbased platforms. The architectures of such SDRs are typically based on heterogeneous multi-processor hardware platforms hosting General Purpose Processors (GPP), Digital Signal Processors (DSP), Field Programmable Gate Arrays (FPGA) and sometimes even more specific processing elements. Each type of processor has a specific advantage in one or the other performance criterion like energy consumption, flexibility, processing time, throughput, waveform portability, components reusability etc. Whenever a new waveform shall be implemented on an SDR, the system designer has to take the decision for the best possible trade-off between all of these, partly conflicting, criteria under the given circumstances. For instance, the portability of waveform components can be improved by fulfilling the constraints of the Software Communications Architecture (SCA) specification and by using common Application Programming Interfaces (API). Thus, the second set of articles puts particular emphasis on topics in the areas of SDR and SCA Implementations.

Reinforcement learning, as a branch of machine learning, has been gradually applied in the control field. However, in the practical application of the algorithm, the hyperparametric approach to network settings for deep reinforcement learning still follows the empirical attempts of traditional machine learning (supervised learning and unsupervised learning). This method ignores part of the information generated by agents exploring the environment contained in the updating of the reinforcement learning value function, which will affect the performance of the convergence and cumulative return of reinforcement learning. The reinforcement learning algorithm based on dynamic parameter adjustment is a new method for setting learning rate parameters of deep reinforcement learning. Based on the traditional method of setting parameters for reinforcement learning, this method analyzes the advantages of different learning rates at different stages of reinforcement learning and dynamically adjusts the learning rates in combination with the temporal-difference (TD) error values to achieve the advantages of different learning rates in different stages to improve the rationality of the algorithm in practical application. At the same time, by combining the Robbins–Monro approximation algorithm and deep reinforcement learning algorithm, it is proved that the algorithm of dynamic regulation learning rate can theoretically meet the convergence requirements of the intelligent control algorithm. In the experiment, the effect of this method is analyzed through the continuous control scenario in the standard experimental environment of ”Car-on-The-Hill” of reinforcement learning, and it is verified that the new method can achieve better results than the traditional reinforcement learning in practical application. According to the model characteristics of the deep reinforcement learning, a more suitable setting method for the learning rate of the deep reinforcement learning network proposed. At the same time, the feasibility of the method has been proved both in theory and in the application. Therefore, the method of setting the learning rate parameter is worthy of further development and research.

Recently, reinforcement learning has made remarkable achievements in the fields of natural science, engineering, medicine and operational research. Reinforcement learning addresses sequence problems and considers long-term returns. This long-term view of reinforcement learning is critical to find the optimal solution of many problems. The existing multi- agent reinforcement learning algorithms have the problem of overestimation in estimating the Q value. Unfortunately, there have not been many studies on overestimation of agent reinforcement learning, which will affect the learning efficiency of reinforcement learning. Based on the traditional multi-agent reinforcement learning algorithm, this paper improves the actor network and critic network, optimizes the overestimation of Q value and adopts the update delayed method to make the actor training more stable. In order to test the effectiveness of the algorithm structure, the modified method is compared with the traditional MADDPG, DDPG and DQN methods in the simulation environment.

Highlights Deep reinforcement learning-based irrigation scheduling is proposed to determine the amount of irrigation required at each time step considering soil moisture level, evapotranspiration, forecast precipitation, and crop growth stage. The proposed methodology was compared with traditional irrigation scheduling approaches and some machine learning based scheduling approaches based on simulation. Abstract. Machine learning has been widely applied in many areas, with promising results and large potential. In this article, deep reinforcement learning-based irrigation scheduling is proposed. This approach can automate the irrigation process and can achieve highly precise water application that results in higher simulated net return. Using this approach, the irrigation controller can automatically determine the optimal or near-optimal water application amount. Traditional reinforcement learning can be superior to traditional periodic and threshold-based irrigation scheduling. However, traditional reinforcement learning fails to accurately represent a real-world irrigation environment due to its limited state space. Compared with traditional reinforcement learning, the deep reinforcement learning method can better model a real-world environment based on multi-dimensional observations. Simulations for various weather conditions and crop types show that the proposed deep reinforcement learning irrigation scheduling can increase net return. Keywords: Automated irrigation scheduling, Deep reinforcement learning, Machine learning.

We present a GPU-based hybrid model for crowd simulations. The model uses reinforcement learning to guide groups of pedestrians towards a goal while adapting to environmental dynamics, and a cellular automaton to describe individual pedestrians’ interactions. In contrast to traditional multi-agent reinforcement learning methods, our model encodes the learned navigation policy into a navigation map, which is used by the cellular automaton’s update rule to calculate the next simulation step. As a result, reinforcement learning is independent of the number of agents, allowing the simulation of large crowds. Implementation of this model on the GPU allows interactive simulations of several hundreds of pedestrians.

Software defined radios (SDR) can be found in every area of wireless signal processing. SDRs realized so far are mostly part of communication technology, including audio broadcast receivers. Image- and video processing in SDRs is not common due to the fact that this type of real-time application requires a huge amount of processing resources. It can be foreseen that SDRs for video processing will be the next attractive approach for flexible and cost-effective media systems. We started with functional diagrams of the required modules for TV receivers, selected suitable algorithms to estimate their computational complexity and investigated their resource-quality scaling properties. Thereafter benchmark tests were done for a variety of common general purpose processors (GPPs), showing that real-time performance will be possible in the near future.

Cognitive radio (CR) technology is a potential solution to the spectrum scarcity problem. In CR systems, the availability of the licensed spectrum portion is monitored by the spectrum sensing process and strategies are applied to use this licensed portion without interfering with active primary users (PU). CR provides a flexible system that secondary users (SU) can make decisions about the spectrum usage at any time by simply configuring corresponding transmission parameters. However, implementation of CR systems can be a challenging task due to characteristic difficulties of the wireless channels introduced by fading. Especially spectrum sensing process is affected by changing channel conditions, which should be considered in order to create a high performance CR system. Robust spectrum sensing is essential for a CR system due to its vital role in the efficient usage of the spectrum. Therefore, several algorithms that are proposed for this issue should contain suitable properties considering realistic channel conditions. Implementation of these algorithms can be realized through software defined radios (SDRs). SDR is a core component of the CR technology and it allows a practical development process with modification on the software rather than hardware. Thus, SDR based approaches to CR problems are quite effective. In this chapter, the state of the art of CR systems are explained in detail by highlighting essential components of the existing studies. Effective approaches to the implementation using SDR systems are given. Moreover, an energy detection based spectrum sensing implementation for 2.4 GHz ISM band is given as an implementation example and channel based spectrum usage is analyzed by using SDR tools LabVIEW and NI USRP-2921 hardware in real-time.

Software-defined radio (SDR) is widely used in wireless research because it allows communication algorithms to be rapidly developed, tested, and validated over the air (OTA) using flexible hardware and software platforms, enabling rapid prototyping across the physical (PHY) and medium access control (MAC) layers. As wireless environments become increasingly dynamic and adversarial, artificial intelligence (AI) and machine learning (ML) have emerged as promising tools to enhance adaptability and resilience in SDR-based systems, particularly in cognitive radio (CR) and wireless security. However, a gap remains in the literature: few surveys link AI/ML methods with their specific SDR implementations and provide insights into what works or does not work in practice. This survey addresses that gap by focusing on two broad domains: (i) CR, where we trace the evolution from classical SDR prototypes (energy/cyclostationary/eigenvalue sensing, spectrum mobility, MAC design) to AI-enabled approaches (shallow learning, deep networks, and reinforcement learning for access and handoff); and (ii) cybersecurity for SDR-enabled radios, where we synthesize jamming, spoofing, and eavesdropping studies from classical toolchains to AI/ML-based detection and mitigation. We compile representative SDR testbeds and toolkits, providing a structured taxonomy that emphasizes what AI/ML can alleviate (such as manual thresholding and moderate hardware drift) and what remains challenging (including wideband real-time scanning on commodity SDRs, resilience to correlated noise and impairments, as well as the burdens of dataset management and retraining). The outcome is a practical roadmap for AI in SDR, guiding researchers and practitioners from initial concepts to reproducible, hardware-supported experiments.

In deep drawing, reasonable control of blank holder force is the key to the quality of finished products. Traditional blank holder force control methods often need to model the highly non-linear deep drawing process. Because it is difficult to obtain an accurate system dynamics model, the results of traditional methods deviate from the actual situation. In this paper, a blank holder force control model based on the integration of deep reinforcement learning and finite element analysis is proposed. The blank holder force control policy is optimized by combining the perception ability of deep neural network with the decision-making ability of reinforcement learning, which avoiding the fitting of system dynamics. Firstly, an algorithm of blank holder force optimization based on deep reinforcement learning is proposed. The deep neural network is used to deal with the large state space. Secondly, by using a new network structure to construct the policy network, the blank holder force policy is divided into global part and local part, which effectively improves the control effect of policy. Experiments show that the proposed control model can effectively optimize blank holder force control policy and improve product quality compared with the traditional deep reinforcement learning algorithm.

New wireless communications standards do not replace old ones, instead the number of standards keeps on increasing and by now an abundance of standards exists. Moreover there is no reason to assume that this trend will ever stop. Therefore, the software-radio concept is emerging as a potential pragmatic solution: a software implementation of the user terminal able to dynamically adapt to the radio environment in which the terminal is located. For manufacturers this could result in shorter development time, cheaper production due to higher volumes. Furthermore software radio has advantages for consumers because it enables only software updates for new functionality without new hardware. Because of the analog nature of the air interface, a software radio will always have an analog front-end. In an ideal software radio, the analog-to-digital converter (ADC) and the digital-to-analog converter (DAC) are positioned directly after the antenna. Such an implementation is not feasible due to the power that such a device would consume and other physical limitations. It is therefore a challenge to design a system that preserves most properties of the ideal software radio while being realizable with current-day technology. Such a system is called a software-defined radio (SDR). A flexible, all-standard radio will consume more energy than a dedicated radio for a single standard. The first flexible radio could be an application where power consumption is less an issue; an example being a flexible radio in a notebook. This thesis presents such a radio with the focus on wireless LAN standards. These standards use phase modulation or OFDM in the $2.4$ GHz or $5$ GHz frequency band, so we decided in our project to combine a receiver of a phase modulation standard (Bluetooth) with an OFDM receiver (HiperLAN/2) first. In our view SDR is an implementation technology: the HiperLAN/2 hardware is that complex to the Bluetooth hardware that the Bluetooth receiver may be added to the HiperLAN/2 one at limited costs. In this thesis we describe how the physical layer of the OSI model of a phase-modulation receiver can be combined with an OFDM receiver at a functional level. The second goal of this research is to investigate whether this combined receiver can be implemented in software running on a General Purpose Processor (GPP) and estimate the costs of such an implementation with respect to power consumption and computational power requirements. For this reason, a testbed has been developed where all baseband physical layer functions have been successfully mapped on a Pentium 4 processor. This has been tested in combination with a CMOS-integrated wideband analog SDR front-end, containing a low noise amplifier, down-conversion mixers and filters. The testbed can be extended to other standards, because the only limitations in our testbed are the maximal channel bandwidth of $20$ MHz, the dynamic range of the wideband SDR analog front-end and the processing capabilities of the used PC. In 1999, Bose showed for second-generation mobile standards that it is possible to perform most baseband functions of the physical layer on a GPP processor. In his research, I/O was one of the main bottlenecks. In thesis we have identified that this is no longer true for current-day computers. Nowadays, an important bottleneck is the latency of the operating system, because wireless standards require fast response times. Major contributions of this thesis to the field of SDR research are extending the work of Bose for wireless LAN standards and developing a functional SDR receiver architecture that is capable of receiving standards that use OFDM or phase modulation.

Study on deep reinforcement learning techniques for building energy consumption forecasting

Software Defined Radios (SDRs) are wireless communication systems in which many of the radios' important functional elements, including filtering, frequency translation and modulation/demodulation, are implemented in software via Digital Signal Processing (DSP) techniques using embedded processors and Field Programmable Gate Arrays (FPGA) as described by Harris and Lowdermilk in 2010 [1]. SDRs differ from traditional radios that typically employ custom silicon to implement such functions. SDRs exhibit several advantages over traditional radios, including re-configurability and reduced development costs and time-to-market since they do not require custom silicon. SDRs also enjoy long life cycles, since they can readily be adapted to meet future needs. Cognitive Radios (CRs) are a special class of SDRs that employ adaptive DSP technologies to implement spectral management. This enables CRs to deliver improved effective bandwidth, quality of service, and enhanced security, which have become increasingly important in modern commercial and military wireless communications applications.

Reinforcement learning is an unsupervised learning method which has been used in many fields. Actually, the essence of Reinforcement learning is a decision-making problem. It constantly tries to interact with the environment. Each interaction process will get a different feedback value, and then it adjusts each trial strategy through feedback. In this paper, we apply the Reinforcement learning technology to software defined network routing algorithm, and propose the routing algorithm based on Q-learning. Through the combination of Reinforcement learning and neural network, which means the Q-table in Q-learning is replaced by neural network, we present routing algorithm based on Deep Q-learning.

Reinforcement learning in neural nets is an approach to the problem of credit assignment during learning. As opposed to gradient descent techniques such as backpropagation, a reinforcement learning scheme uses a single reinforcement signal from the environment to adjust the network weights. In this short paper we describe reinforcement learning and propose a multilayer neural network with real-valued outputs which learns using a combination of reinforcement learning and backpropagation. This method combines several ideas from the literature. We illustrate the use of the method with an example of the control of a nonlinear system.

Recently, machine learning has been very useful in solving diverse tasks with drones, such as autonomous navigation, visual surveillance, communication, disaster management, and agriculture. Among these machine learning, two representative paradigms have been widely utilized in such applications: supervised learning and reinforcement learning. Researchers prefer to use supervised learning, mostly based on convolutional neural networks, because of its robustness and ease of use but yet data labeling is laborious and time-consuming. On the other hand, when traditional reinforcement learning is combined with the deep neural network, it can be a very powerful tool to solve high-dimensional input problems such as image and video. Along with the fast development of reinforcement learning, many researchers utilize reinforcement learning in drone applications, and it often outperforms supervised learning. However, it usually requires the agent to explore the environment on a trial-and-error basis which is high cost and unrealistic in the real environment. Recent advances in simulated environments can allow an agent to learn by itself to overcome these drawbacks, although the gap between the real environment and the simulator has to be minimized in the end. In this sense, a realistic and reliable simulator is essential for reinforcement learning training. This paper investigates various drone simulators that work with diverse reinforcement learning architectures. The characteristics of the reinforcement learning-based drone simulators are analyzed and compared for the researchers who would like to employ them for their projects. Finally, we shed light on some challenges and potential directions for future drone simulators.