Towards Concept Based Software Engineering for Intelligent Agents

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement–</i> Significant performance milestones in reinforcement learning have been achieved in recent years, with autonomous agents demonstrating super-human performance across a wide variety of tasks. Before these algorithms can be extensively deployed in real-world defense applications, a greater level of trust must first be achieved. ADT was an important step towards developing the trust necessary to operationalize these algorithms, by demonstrating their effectiveness on a foundational yet relevant problem in a high-fidelity simulation environment. Developed for the program, our hierarchical reinforcement learning agent was designed alongside of and competed against active fighter pilots, and ultimately defeated a graduate of the United States Air Force's F-16 Weapons Instructor Course in match play.

Recently, some challenging tasks in multi-agent systems have been solved by some hierarchical reinforcement learning methods. Inspired by the intra-level and inter-level coordination in the human nervous system, we propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for fully cooperative multi-agent problems. To address the instability arising from the concurrent optimization of policies between various levels and agents, we introduce the dual coordination mechanism of inter-level and inter-agent strategies by designing reward functions in a two-level hierarchy. HAVEN does not require domain knowledge and pre-training, and can be applied to any value decomposition variant. Our method achieves desirable results on different decentralized partially observable Markov decision process domains and outperforms other popular multi-agent hierarchical reinforcement learning algorithms.

Theory-oriented software engineering

In this paper, a method that utilizes the reinforcement learning (RL) technique is proposed to establish an optimal operation plan to obtain maximum power output from a trigen generator. Trigen is a type of combined heat and power system (CHP) that provides chilling, heating, and power generation, and the turbo expander generator (TEG) is a generator that uses the decompression energy of gas to generate electricity. If the two are combined to form a power source, a power generation system with higher efficiency can be created. However, it is very difficult to control the heat and power generation amount of TEG and trigen according to the flow rate of natural gas that changes every moment. Accordingly, a method is proposed to utilize the RL technique to determine the operation process to attain an even higher efficiency. When the TEG and trigen are configured using the RL technique, the power output can be maximized, and the power output variability can be reduced to obtain high-quality power. When using the RL technique, it was confirmed that the overall efficiency was improved by an additional 3%.

The recent development of technology helps in the revolutionary war and it controls the war which is influenced by brilliant planning. The maneuver aircraft of intelligent algorithm aids the pilot to decide the particular position on the battlefield. Nowadays the hardware components of radar and missiles are widely used and the beyond visual range is the most popular method applied in air combat. The introduction of close-range air combat maneuver decisions generates the attention of researchers in artificial intelligence. Most of the existing methods are based on autonomous aircraft focused in air combat scenario but manual air combats are widely applied in dual aircraft. Based on the factors mentioned above, a novel hierarchical maneuver decision architecture is applied to a dual-aircraft close-range air combat scenario. Subsequently, the soft actor-critic algorithm is merged with competitive self-play which integrates the knowledge of sub-strategies. Further, the reinforcement learning technique is employed to achieve an approximate Nash equilibrium master strategy. The experimental results show that the hierarchical architecture exhibits good performance, symmetry, and robustness. The research generates a solution for intelligent formation of air combat in the future and guidance for manned or unmanned aircraft cooperative combat.

In this work, we analyze and improve upon reinforcement learning techniques used to build agents that can learn to play Infinite Mario, an action game. We extend the object-oriented representation by introducing the concept of object classes which can be effectively used to constrain state spaces. We then use this representation combined with the hierarchical reinforcement learning model as a learning framework. We also extend the idea of hierarchical RL by designing a hierarchy in action selection using domain specific knowledge. With the help of experimental results, we show that this approach facilitates faster and efficient learning for this domain.

In recent years, the use and investigations about Grey Literature (GL) increased, in particular, in Software Engineering (SE) research. However, its understanding is still scarce and sometimes controversial, such as interpreting GL types and assessing their credibility. This study aimed to understand the credibility aspects that SE researchers consider in assessing GL and its types. To achieve this goal, we surveyed 53 SE researchers (who answered that they have used GL in our previous investigation), receiving a total of 34 valid responses. Our main findings show that: 1) GL source produced or cited by a renowned source is the main credibility criteria used to assess GL, 2) most of the GL types tend to have a Low to Moderate level of Control and Expertise, 3) there is a positive statistical correlation between the level of Control and Expertise for most GL types, and 4) the different respondent profiles shared similar opinions about the credibility criteria. Our investigation contributes to helping future SE researchers that intend to use GL with more credibility. Additionally, shows the need for future studies to better understand the GL types in SE research.

Hierarchical reinforcement learning has achieved good results in solving complex learning tasks. Hierarchical reinforcement learning mainly includes on-policy and off-policy methods. On-policy cannot be applied to actual scenarios due to low data utilization. Therefore, off-policy reinforcement learning methods have become the main development direction. However, in the off-policy method, because the data in the replay buffer comes from different policy, the upper-layer policy gives the same goal, and the lower-layer policy is constantly updated and shifts to a different state, so the upper-layer policy cannot be stably trained. Aiming at the above problems, we propose a hierarchical reinforcement learning algorithm for environment representation based on the mutual bisimulation Metrics. When training the upper policy, the lower policy is used as a part of the environment, which is called virtual representation environment. The output is used for feature extraction, and the feature value is used as the state value of the upper-level policy net. Using our proposed method to compare a variety of complex tasks with the current main hierarchical reinforcement learning has achieved the stability and training effect improvement.

Vehicle routing problem with time windows (VRPTW) is a practical and complex vehicle routing problem (VRP) which is faced by thousands of companies in logistics and transportation. Usually, VRP is solved by traditional heuristic algorithms. Recently, deep learning models under the reinforcement learning (RL) framework have been proposed to solve variants of VRP. In our study, we propose to use the hierarchical RL to find an optimal policy for generating optimal routes in VRPTW. The hierarchical RL structure includes a low level which generates feasible solutions and a high level which further searches for an optimal solution. Experimental results show that the proposed hierarchical RL model outperforms the non-hierarchical RL model and the heuristic algorithms Google OR-Tools. The proposed model also shows generalization capability in three different scenarios: varied time window constraints, from small-scale to large-scale problems, and generalization across different datasets. The flexible framework of hierarchical RL can also be applied to solve other complex VRPs with multiple objectives.

Heuristic production control policies such as CONWIP, kanban, and other hybrid policies have been in use for years as better alternatives to MRP-based push control policies. It is a fact that these policies, although efficient, are far from optimal. Our goal is to develop a methodology that, for a given system, finds a dynamic control policy via intelligent agents. Such a policy while achieving the productivity (i.e., demand service rate) goal of the system will optimize a cost/reward function based on the WIP inventory. To achieve this goal we applied a simulation-based optimization technique called Reinforcement Learning (RL) on a four-station serial line. The control policy attained by the application of a RL algorithm was compared with the other existing policies on the basis of total average WIP and average cost of WIP. We also develop a heuristic control policy in light of our experience gained from a close examination of the policies obtained by the RL algorithm. This heuristic policy named Behavior-Based Control (BBC), although placed second to the RL policy, proved to be a more efficient and leaner control policy than most of the existing policies in the literature. The performance of the BBC policy was found to be comparable to the Extended Kanban Control System (EKCS), which as per our experimentation, turned out to be the best of the existing policies. The numerical results used for comparison purposes were obtained from a four-station serial line with two different (constant and Poisson) demand arrival processes.

This article covers the strategic, organizational, and operational decisions involved in one automobile manufacturer's efforts to balance its goals of productivity and high quality with the more elusive goal of environmental responsibility. The case material is based on the real‐life experiences of Honda of America's two manufacturing plants in East Liberty and Marysville, Ohio. The case unfolds as the environmental manager in charge of the two plants faces the pending visit of her corporate boss from Tokyo, who has made it clear that environmental issues are of growing importance in Honda's overall direction.ISO 14000, the new environmental quality standards being adopted by some other manufacturers, serves as a focusing issue for the environmental manager's thinking, but this issue is really representative of a wider challenge facing the Ohio plants: balancing Honda's core mission of producing the best cars at the lowest cost possible, and being a responsible actor in the environmental arena.Complicating this challenge, the environmental manager has to negotiate another tension common in global manufacturing firms today: balancing the influence of Honda's corporate headquarters in Japan with the local and regional context at its American plant sites.

Autonomous systems need to be able dynamically adapt to changing requirements and environmental conditions without redeployment and without interruption of the systems functionality. The EU project ASCENS has developed a comprehensive suite of foundational theories and methods for building autonomic systems. In this paper we specialise the EDLC process model of ASCENS to deal with planning and reinforcement learning techniques. We present the “AIDL” life cycle and illustrate it with two case studies: simulation-based online planning and the PSyCo reinforcement learning approach for synthesizing agent policies from hard and soft requirements. Related work and potential avenues for future research are discussed.

Background: The use of Grey Literature (GL) has been investigated in diverse research areas. In Software Engineering (SE), this topic has an increasing interest over the last years. Problem: Even with the increase of GL published in diverse sources, the understanding of their use on the SE research community is still controversial. Objective: To understand how Brazilian SE researchers use GL, we aimed to become aware of the criteria to assess the credibility of their use, as well as the benefits and challenges. Method: We surveyed 76 active SE researchers participants of a flagship SE conference in Brazil, using a questionnaire with 11 questions to share their views on the use of GL in the context of SE research. We followed a qualitative approach to analyze open questions. Results: We found that most surveyed researchers use GL mainly to understand new topics. Our work identified new findings, including: 1) GL sources used by SE researchers (e.g., blogs, community website); 2) motivations to use (e.g., to understand problems and to complement research findings) or reasons to avoid GL (e.g., lack of reliability, lack of scientific value); 3) the benefit that is easy to access and read GL and the challenge of GL to have its scientific value recognized; and 4) criteria to assess GL credibility, showing the importance of the content owner to be renowned (e.g., renowned author and institutions). Conclusions: Our findings contribute to form a body of knowledge on the use of GL by SE researchers, by discussing novel (some contradictory) results and providing a set of lessons learned to both SE researchers and practitioners.

This literature review research paper examines the application of AI and machine learning in the financial industry and its effects on risk management and fraud detection. The study conducts a comprehensive search of academic and industry sources and identifies key findings and trends related to the use of these technologies in the financial industry. The literature review finds that AI-based systems have been shown to improve the efficiency and effectiveness of fraud detection by analysing vast amounts of data in real-time and identifying patterns and anomalies that may indicate potential fraud (KPMG, 2018). Additionally, machine learning algorithms can be trained to adapt and improve over time, making the detection process even more accurate. Furthermore, AI-based systems have also been shown to improve risk management in the financial industry (Deloitte, 2019). However, it is important to note that while AI and machine learning have the potential to significantly improve risk management and fraud detection in the financial industry, they are not without limitations and potential biases (Bolton and Hand, 2002; Bose, 2018). Therefore, financial institutions should carefully consider the implementation and use of these technologies and ensure proper governance and controls are in place. This literature review provides a comprehensive understanding of the current state of AI and machine learning in the financial industry, its benefits and challenges, and future possibilities. Keywords: AI, Machine Learning, Financial Industry, Risk Management, Fraud Detection, Governance, Ethical Implications

Sequential decision-making over long timescales and in complex task environments is an important problem in Artificial Intelligence (AI). An effective approach to tackle this problem is to autonomously decompose a long-horizon task into a sequence of simpler subtasks or subgoals. We refer to this approach as Autonomous Task Decomposition (ATD) in the thesis and study it for multi-agent coordination using model-free Hierarchical Reinforcement Learning (HRL), single-agent goal-reaching using model-free HRL, and single-agent goal-reaching using model-based planning. The objective of the thesis is to develop novel methods to address three important challenges related to ATD, which are as follows: 1. Effective multi-agent HRL under sparse global rewards and complex inter-dependencies among agents. 2. Efficient unification of autonomous subgoal discovery and single-agent HRL without slow learning. 3. Learning models for planning-based ATD that produce more rewarding and feasible plans. In this regard, the thesis introduces three novel ATD methods as follows: 1. Inter Subtask Empowerment based Multi-agent Options (ISEMO) is introduced for effective multi-agent HRL by using auxiliary rewards that capture the inter-dependencies among HRL agents and their (handcrafted) subtasks. ISEMO leads to better coordinated performance of the inter-dependent agents on a complex Search & Rescue task, compared to a standard multi-agent HRL method. 2. End-to-End Hierarchical Reinforcement Learning with Integrated Discovery of Salient Subgoals (LIDOSS) is introduced for efficient unification of subgoal discovery and HRL for single-agent goal-reaching, by using a probability-based subgoal discovery heuristic integrated with the subgoal selection policy. LIDOSS accelerates end-to-end learning and leads to higher goal-reaching success rates compared to a state-of-the-art HRL method. 3. Finally, Learning Subgoal Graph using Value-based Subgoal Discovery and Automatic Pruning (LSGVP) is introduced to learn subgoal graph-based planning models that produce more rewarding and feasible plans for single-agent goal-reaching. LSGVP uses cumulative reward-based subgoal discovery and automatic pruning of erroneous connections in the subgoal graph. It achieves higher positive cumulative rewards and higher success rates compared to other state-of-the-art subgoal graph-based planning methods, while also being more data-efficient than model-free HRL.