Introduction

The purpose of this section is to introduce the notation that will be used in the subsequent parts and the most essential facts that we will need from the theory of Markov Decision Processes (MDPs) in the rest of the book. Readers familiar with MDPs should skim through this section to familiarize themselves with the notation. Readers unfamiliar with MDPs are suggested to spend enough time with this section to understand the details. Proofs of most of the results (with some simplifications) are included in Appendix A. The reader who is interested in learning more about MDPs is suggested to consult one of the many excellent books on the subject, such as the books of Bertsekas and Shreve (1978), Puterman (1994), or the two-volume book by Bertsekas (2007a,b).

In this paper, we propose an optimal video streaming framework using retransmissions based on the theory of Markov decision processes (MDP). Using MDPs, we calculate average reward optimal policies so as to maximize the on time delivery probability of video frames in the transmission window while conforming to network-imposed available bandwidth constraints. The effectiveness of the proposed approach is validated by comparing the results with theoretically obtained rate-distortion curves corresponding to the optimal policies by using the distortion model in K. Stuhlmuller et al., (2000)

We consider in this study dynamic control policies for slotted Aloha random access systems. New performance bounds are derived when random access is combined with power control for system optimization, and we establish the existence of optimal control approaches for such systems. We analyze throughput and delay when the number of backlogged users is known, where we can explicitly obtain optimal policies and analyze their corresponding performance using Markov Decision Process (MDP) theory with average cost criterion. For the realistic unknown-backlog case, we establish the existence of optimal backlog-minimizing policies for the same range of arrival rates as the ideal known-backlog case by using the theory of MDPs with Borel state space and unbounded costs. We also propose suboptimal control policies with performance close to the optimal without sacrificing stability. These policies perform substantially better than existing Certainty Equivalence controllers.

We consider the problem of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">quickest</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">detection</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">of</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">an</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">intrusion</i> using a sensor network, keeping only a minimal number of sensors <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">active</i> . By using a minimal number of sensor devices, we ensure that the energy expenditure for <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sensin</i> g, <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">computation</i> and <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">communication</i> is minimized (and the lifetime of the network is maximized). We model the intrusion detection (or change detection) problem as a <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Markov</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">decision</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">process</i> (MDP). Based on the theory of MDP, we develop the following closed loop <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sleep/wake</i> scheduling algorithms: (1) optimal control of M <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</sub> , the number of sensors in the wake state in time slot k + 1, (2) optimal control of q <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</sub> , the probability of a sensor in the wake state in time slot k + 1, and an open loop <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sleep/wake</i> scheduling algorithm which (3) computes q, the optimal probability of a sensor in the wake state (which does not vary with time), based on the sensor observations obtained until time slot k. Our results show that an optimum closed loop control on M <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</sub> significantly decreases the cost compared to keeping any number of sensors active all the time. Also, among the three algorithms described, we observe that the total cost is minimum for the optimum control on M <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</i> and is maximum for the optimum open loop control on q.

This paper presents an optimization model for selection of pavement maintenance intervention using a theory of Markov Decision Processes (MDP). There are some particular characteristics of the MDP developed in this paper which distinguish it from other similar studies or optimization models intended for pavement maintenance policy development. These unique characteristics include a direct inclusion of constraints into the formulation of MDP, the use of an average cost method of MDP, and the policy development process based on the dual linear programming solution. The limited information or discussions that are available on these matters in terms of stochastic based optimization model in road network management motivates this study. This paper uses a data set acquired from road authorities of state of Victoria, Australia, to test the model and recommends steps in the computation of MDP based stochastic optimization model, leading to the development of optimum pavement maintenance policy.

It was a great pleasure to welcome all the participants from different countries to the 2021 6th International Symposium on Advances in Electrical, Electronics and Computer Engineering (ISAEECE 2021) in Nanjing, China during March 12-14, 2021. ISAEECE 2021 is to bring together innovative academics and industrial experts in the field of electrical, electronics and computer engineering to a common forum. The primary goal of the conference is to promote research and developmental activities in electrical, electronics and computer engineering and another goal is to promote scientific information interchange between researchers, developers, engineers, students, and practitioners working all around the world. The conference will be held every year to make it an ideal platform for people to share views and experiences in electrical, electronics and computer engineering and related areas.The conference model was divided into two sessions, including oral presentations and keynote speeches. In the first part, some scholars, whose submissions were selected as the excellent papers, were given 15 minutes to perform their oral presentations one by one. Then in the second part, keynote speakers were each allocated 30-45 minutes to hold their speeches. In this conference, we were greatly honor to have invited four outstanding researchers to serve as our Conference Chairman.During the conference, we invited three industry elites to present their insightful speeches. The first keynote speaker, Dr. Sunil Kumar Jha, from Nanjing University of Information Science and Technology, China. He presented a speech: Data Fusion Approaches in Human Body Odor Data Mining. The objective of the present research talk is to introduce audience about the data fusion and human body odor and to demonstrate research results related to search for an optimal subset of VOCs in body odor, which can produce differentiation in an individual by using the combination of analytical methods and chemometric analysis. The second keynote speaker, Prof. Chunbo Xiu, from Tiangong University, China. He shared a speech on his specialty related to Memristive Cellular Neural Network & Its Dynamic Characteristic Analysis. He demonstrated that chaotic memristive CNN system can be used in the secure communication by the chaos synchronization based on sliding mode control. Assoc. Prof. Pavel Loskot, from Zhejiang University-University of Illinois at Urbana-Champaign Institute (ZJUI), our third keynote speakers. He presented a speech: Monte Carlo Simulations Revisited. Their insightful speeches made every participant praised this conference for disseminating useful knowledge.We were glad to share with you that we received lots of submissions from the conference and we selected a bunch of high-quality papers and compiled them into the proceedings after rigorously reviewed them. These papers feature following topics but are not limited to: Electronics Engineering, Electrical Engineering, Computer Engineering and other related topics. All the papers have been through rigorous review and process to meet the requirements of International publication standard.We were grateful to the International/National advisory committee, keynote speakers, session chairs, organising committee members, student volunteers and administrative assistance of the management section of University, including accounts section, digital media and publication house. Also, we were thankful to all the authors for contributing large number of papers in the conference, because of which the conference became a story of success. It was the quality of their presentations and their passion to communicate with the other participants that really make this conference series a great success.The Organizing Committee of ISAEECE 2021List of Committee member are available in this pdf.

Due to the limited availability of corpora in the field of Electrical Engineering and the presence of numerous specialized terms, neural machine translation (NMT) performs poorly in translating the sentence backbone information when it is applied to corpora in the field of Electrical Engineering. In response to this issue, a method to improve NMT by using the sentence backbone information is proposed in this paper. In the proposed method, the source language sentences are used as the input of the Sentence Backbone Information Extraction Model to obtain the sentence backbone information, and then the sentence backbone information are incorporated as an auxiliary during the training process of the NMT model. Furthermore, a module called the Sentence Backbone Information Enhancement Module is introduced. It utilizes the dependency parse trees of the source language sentences to generate the sentence backbone mask matrices. These matrices are then applied to the encoder to force the NMT model to pay more attention to the backbones of sentences. On the English-Chinese parallel corpus in the field of Electrical Engineering, the proposed method in this paper outperforms the Transformer baseline translation model by 1.25 BLEU points. And it outperforms the baseline model in both METEOR and ROUGE-L evaluation metrics. It indicates that the proposed method in this paper can effectively improve translation performance.

We present our process and development of a web-based system to explore the publication networks of faculty in California public universities in the fields of computer science and electrical engineering. Our project explores collaboration networks in the fields of computer science and electrical engineering with a focus on publication networks and an analysis of these collaborations with a focus on geospatial organization (which institutions are collaborating with which other institutions). We present our data gathering process, which relies on the Scopus[9] database (Scopus represents a “comprehensive overview of the world’s scientific research output across all disciplines”), and we present the development of a web-based tool using python and the Google Maps API [12] in order to allow visualizations and explorations of geospatial structures of the publications networks. These visualizations drove further network analysis, which we present here as well.

Abstract: This paper presents a case study of designing an e-learning course in the field of electrical engineering- Industrial automation program at Palestine Polytechnic University. The applied methodology consisting of few stages starting with formulating the major course objectives, extracting topics outcomes, design the appropriate teaching tools and storyboard, and as final stage the evaluation system is designed.&#x0D; On other hand, an information communication technology (ICT) questionnaire has been designed and targeting attended in the course students with purpose evaluating student ICT knowledge, knowledge leveling, and treatment their weaknesses.&#x0D; The designed topics are modified according to feedback evaluation obtained from students throughout designed for this purpose questionnaire. The observed weaknesses in course topics have been eliminated and finally the modified course was uploaded to the university e-learning platform. Electrical engineering core course with 300 level was carried out for department students, where the obtained results shows significant enhancement in teaching and learning performances of both course instructors and students.

Chapter 1 Applications of network optimization

This chapter presents an overview of simulation-based techniques useful for solving Markov decision processes (MDPs). MDPs model problems of sequential decision-making under uncertainty, in which decisions made in each state collectively affect the trajectory of the states visited by the system over a time horizon of interest. Traditionally, MDPs have been solved via dynamic programming (DP), which requires the transition probability model that is difficult to derive in many realistic settings. The use of simulation for solving MDPs allows us to bypass the transition probability model and solve large-scale MDPs considered intractable to solve by traditional DP. The simulation-based methodology for solving MDPs, which like DP is also rooted in the Bellman equations, goes by names such as reinforcement learning, neuro-DP, and approximate or adaptive DP. We begin with a description of algorithms for infinite-horizon discounted reward MDPs, followed by the same for infinite-horizon average reward MDPs. Then we present a discussion on finite-horizon MDPs. For each problem considered, we present a step-by-step description of a selected group of algorithms. In making this selection, we have attempted to blend the old and the classical with more recent developments. Finally, after touching upon extensions and convergence theory, we conclude with a brief summary of some applications and directions for future research.

In 1941 the AIEE awarded the Edison Medal to John B. Whitehead for his contributions in the fields of electrical engineering and dielectric research, and for his achievements in engineering education.

This paper studies the admission control and resource allocation in a cell-based wireless system that supports single- media and multirate services. Utilizing the idea of adaptive and dynamic reallocation, we propose three control policies, AO, ADRF, and ADRA. The Markov decision process (MDP) theory is used to build the model of the system. We solve the MDP model and find the optimal policy by linear programming (LP). The proposed policies not only make the best use of precious wireless bandwidth, but also satisfy the requirement of quality of services such as handover blocking probability.

My research in optimization at the Department of Systems and Computer Engineering at Carleton University has brought me into contact with researchers in the Department of Electronics, our sister department in the general field of Electrical and Computer Engineering (ECE), which has a major research thrust in computer-aided design (CAD) of very large scale integrated (VLSI) circuits. I have come to realize that the design of VLSI systems consists in large part of solving incredibly massive mixed-integer nonlinear optimization problems, together with enormous circuit simulations. This is impossible without the use of CAD techniques. Over time, ECE-CAD researchers have both borrowed useful standard techniques from operations research (OR) and have invented their own, often to deal with the sheer scale and complexity of the design problems they face. Interaction between the OR and ECE-CAD communities would seem to be a natural development. Coincidentally, the INFORMS Journal on Computing had an area entitled “High Performance Computation” that covered (i) the solution of OR problems using new computing technologies, (ii) the application of OR techniques in the design and use of highperformance computing and communication systems, and (iii) solution methods for ultra-large-scale OR applications. The viability of this area was debated during 2003 as some of its aspects migrated to other areas (e.g., the new “Telecommunications and Electronic Commerce” area), and the number of submissions declined. The area was eventually closed, although I argued that the overlap between OR and ECE-CAD, essentially covering items (ii) and (iii), was a very active research area. This Special Cluster of papers was conceived as a way to test that argument. I was recruited at the same time to prepare a tutorial on ECE-CAD for the 2004 INFORMS Annual Meeting (see John W. Chinneck, Michel Nakhla, and Q. J. Zhang 2004. Computer-aided design for electrical and computer engineering. H. J. Greenberg, ed. Tutorials on Emerging Methodologies and Applications in Operations Research. Springer, New York, 6-1 to 6-44). The preparation of the article reinforced my observation about the research overlap between OR and ECE-CAD. A search of the electrical engineering literature for 2000 through early 2004 turned up 46,725 papers mentioning “simulate” or “simulation” as keywords in the abstract, 14,216 papers mentioning “optimization” or “optimize,” and relatively smaller numbers for specific techniques such as “neural network” (6,251), “genetic algorithm” (2,603), “linear programming” (576), “simulated annealing” (448), and “branch and bound” (199). One surprise was the relatively small number of papers using the generic keywords “mathematical programming” (68) or “operations research” (37). The general conclusion of the tutorial article is that the OR and the ECE-CAD communities have much to offer each other. This is certainly the case for the papers gathered in this Special Cluster. We see known OR techniques adapted for use in ECECAD. In “Integer Linear Programming Models for Global Routing,” Behjat et al. apply integer linear programming in a heuristic to solve enormous NPhard connection routing problems for VLSI circuits. In “Task Scheduling in a Finite-Resource, Reconfigurable Hardware/Software Codesign Environment,” Loo and Wells use simulated annealing, genetic algorithms, and random search techniques to solve scheduling problems in hardware-software co-design. We also see new techniques specifically developed by the ECE-CAD community to deal with problems of extreme scale. In “A Projection-Based Reduction Approach to Computing Sensitivity of Steady-State Response of Nonlinear Circuits,” Pai et al. develop methods for sensitivity analysis in extremely large nonlinear programs, especially those in which the objective function is very costly to evaluate. The methods have special relevance for simulation-based

Development of new technologies, especially Internet, creates conditions for electronic education. The paper presents effect of using Internet as one of the means for taking exams or colloquia in certain subjects in the field of electrical and computer engineering at the Academy of Applied Studies of Kosovo and Metohija, Department Uroševac – Leposavić. Namely, the idea for writing this paper arose due to the fact that, to some extent, this type of communication with students already existed. After taking certain subjects in the field of electrical and computer engineering, research of students was done through a questionnaire. For purposes of this research, a survey questionnaire was used, which included a sample of 58 students. The goal of this research is to examine stances of students on perceived usefulness of the platform and its upgrade for taking exams or colloquia and for other scientific areas on all study programs used in the teaching process. At the end, obtained results were analyzed, based on which conclusion was drawn that students are satisfied with this type of communication as well as with quality of the platform itself.