Keddah

As a distributed system, Hadoop heavily relies on the network to complete data processing jobs. While Hadoop traffic is perceived to be critical for job execution performance, the actual behaviour of Hadoop network traffic is still poorly understood. This lack of understanding greatly complicates research relying on Hadoop workloads. In this paper, we explore Hadoop traffic through experimentation. We analyse the generated traffic of multiple types of MapReduce jobs, with varying input sizes, and cluster configuration parameters. As a result, we present Keddah, a toolchain for capturing, modelling and reproducing Hadoop traffic, for use with network simulators. Keddah can be used to create empirical Hadoop traffic models, enabling reproducible Hadoop research in more realistic scenarios.

Network emulations are widely used for testing novel network protocols and routing algorithms in realistic scenarios. Up to now, there is no emulation tool that is able to emulate large software-defined data center networks that consist of several thousand nodes. Mininet is the most common tool to emulate Software-Defined Networks of several hundred nodes. We extend Mininet to span an emulated network over several physical machines, making it possible to emulate networks of several thousand nodes on just a handful of physical machines. This enables us to emulate, e.g., large data center networks. To test this approach, we additionally introduce a traffic generator for data center traffic. Since there are no data center traffic traces publicly available we use the results of two recent traffic studies to create synthetic traffic. We show the design and discuss some challenges we had in building our traffic generator. As a showcase for our work we emulated a data center consisting of 3200 hosts on a cluster of only 12 physical machines. We show the resulting workloads and the trade-offs involved.

Cloud infrastructures are generally over-provisioned for handling load peaks and node failures. However, the drawback of this approach is that a large portion of data center resources remains unused. In this paper, we propose a framework that leverages unused resources of data centers, which are ephemeral by nature, to run MapReduce jobs. Our approach allows: i) to run efficiently Hadoop jobs on top of heterogeneous Cloud resources, thanks to our data placement strategy, ii) to predict accurately the volatility of ephemeral resources, thanks to the quantile regression method, and iii) for avoiding the interference between MapReduce jobs and co-resident workloads, thanks to our reactive QoS controller. We have extended Hadoop implementation with our framework and evaluated it with three different data center workloads. The experimental results show that our approach divides Hadoop job execution time by up to 7 when compared to the standard Hadoop implementation.

: Three research topics have been investigated: (I) Energy aware programming techniques for embedded systems; (2) Dynamic power management of a sensor node with periodic incoming tasks; and (3) Resource management of a sensor network using a distributed Genetic Algorithm. The research shows that (I) Certain simple program transformations can significantly improve the performance and energy dissipation of the embedded processor, which is the core part of a sensor node or a mobile computing device; (2) The traditional task scheduling algorithm does not work well with power management in a sensor node. Two heuristic algorithms are proposed which give lower power consumption than the traditional algorithms. (3) The power management problem in a distributed system may be formulated as an optimization problem and solved using a distributed GA. The performance and energy of a distributed GA is determined by its configuration parameters such as: the sub-population size, the number of processors, the length of epoch, and the number of migrating individuals. Their relations with the convergence speed and the energy dissipation is analyzed in this report. This effort leveraged a distributed (parallel) Island Model Genetic Algorithm code developed in IFTO that was adapted to the 3rd problem and run on a cluster computer in FTC this summer. A follow-on grant will study a fuller version of the problems with more realistic scenarios.

Simulation-based testing has emerged as an essential tool for verifying and validating autonomous vehicles (AVs). However, contemporary methodologies, such as deterministic and imitation learning-based driver models, struggle to capture the variability of human-like driving behavior. Given these challenges, we propose HAD-Gen, a general framework for realistic traffic scenario generation that simulates diverse human-like driving behaviors. The framework first clusters the vehicle trajectory data into different driving styles according to safety features. It then employs maximum entropy inverse reinforcement learning on each of the clusters to learn the reward function corresponding to each driving style. Using these reward functions, the method integrates offline reinforcement learning pre-training and multi-agent reinforcement learning algorithms to obtain general and robust driving policies. Multi-perspective simulation results in highway scenarios show that our proposed scenario generation framework can generate diverse, human-like driving behaviors with strong generalization capability. The proposed framework achieved a 90.96% goal-reaching rate, an off-road rate of 2.08%, and a collision rate of 6.91% in new unseen driving scenarios, outperforming prior approaches by over 20% in goal-reaching performance. The source code is released at https://github.com/RoboSafe-Lab/HAD-Gen.

The Peer-to-Peer network paradigm is drawing the attention of both final users and researchers for its features. P2P networks shift from the classic client-server approach to a high level of decentralization where there is no central control and all the nodes should be able not only to require services, but to provide them to other peers as well. While on one hand such high level of decentralization might lead to interesting properties like scalability and fault tolerance, on the other hand it implies many new problems to deal with. A key feature of many P2P systems is openness, meaning that everybody is potentially able to join a network with no need for subscription or payment systems. The combination of openness and lack of central control makes it feasible for a user to free-ride, that is to increase its own benefit by using services without allocating resources to satisfy other peers’ requests. One of the main goals when designing a P2P system is therefore to achieve cooperation between users. Given the nature of P2P systems based on simple local interactions of many peers having partial knowledge of the whole system, an interesting way to achieve desired properties on a system scale might consist in obtaining them as emergent properties of the many interactions occurring at local node level. Two methods are typically used to face the problem of cooperation in P2P networks: 1) engineering emergent properties when designing the protocol; 2) study the system as a game and apply Game Theory techniques, especially to find Nash Equilibria in the game and to reach them making the system stable against possible deviant behaviors. In this work we present an evolutionary framework to enforce cooperative behaviour in P2P networks that is alternative to both the methods mentioned above. Our approach is based on an evolutionary algorithm inspired by computational sociology and evolutionary game theory, consisting in having each peer periodically trying to copy another peer which is performing better. The proposed algorithms, called SLAC and SLACER, draw inspiration from tag systems originated in computational sociology, the main idea behind the algorithm consists in having low performance nodes copying high performance ones. The algorithm is run locally by every node and leads to an evolution of the network both from the topology and from the nodes’ strategy point of view. Initial tests with a simple Prisoners’ Dilemma application show how SLAC is able to bring the network to a state of high cooperation independently from the initial network conditions. Interesting results are obtained when studying the effect of cheating nodes on SLAC algorithm. In fact in some cases selfish nodes rationally exploiting the system for their own benefit can actually improve system performance from the cooperation formation point of view. The final step is to apply our results to more realistic scenarios. We put our efforts in studying and improving the BitTorrent protocol. BitTorrent was chosen not only for its popularity but because it has many points in common with SLAC and SLACER algorithms, ranging from the game theoretical inspiration (tit-for-tat-like mechanism) to the swarms topology. We discovered fairness, meant as ratio between uploaded and downloaded data, to be a weakness of the original BitTorrent protocol and we drew inspiration from the knowledge of cooperation formation and maintenance mechanism derived from the development and analysis of SLAC and SLACER, to improve fairness and tackle freeriding and cheating in BitTorrent. We produced an extension of BitTorrent called BitFair that has been evaluated through simulation and has shown the abilities of enforcing fairness and tackling free-riding and cheating nodes.

Most of the models intended to describe the throughput of Primary (PUs) and Secondary (SUs) users of Cognitive Radio Networks (CRNs) assume that PUs only change their activity state (ON/OFF) in the beginning of each SU's operation cycle, admitting that PUs are synchronized with SU's operation cycle. This letter characterizes a more realistic scenario where PUs can randomly change their activity state during the SU's operation cycle. We derive an analytical model for the PU's throughput and its validation is assessed through simulation results. The analysis shows that assuming synchronized PUs leads to an undervaluation of the interference caused to PUs, and the interference decreases as more SU's operation cycles are performed per ON/OFF PU's activity state.

The main objective of emergent standard Virtual Private LAN Service (VPLS) is the interconnection of users located at different geographic locations as if they were in the same local area network. In this sense, the investigations and tests that have been done are mainly focused on point-to-point interconnection scenarios using Virtual Private Networks (VPNs), as there have been few tests done up until now in academic and research networks with multipoint scenarios and they have been done mainly within the same domain, but not between different domains. In this paper, we describe the process to put in production VPLS service for testing purposes in an interprovider environment, doing initial tests on a real scenario. With it, one hopes to verify its behaviour in a real production network, analysing the benefits that can be provided to the customers, especially with respect to advanced applications and distributed computing, as it is the case in Grid applications.

Intelligent Transport Systems (ITS) are now being implemented to ensure optimal and safe road traffic. Increasingly, these systems use artificial intelligence to obtain characteristics about traffic flows. The number of sensors and transducers is increasing dramatically, resulting in higher loads on ITSs. Therefore, it is necessary to develop distributed monitoring systems with scalability and fault tolerance in mind. However, extensive testing is required before implementation. It is not possible to fully conduct such testing on real data due to various factors. Therefore, this paper proposes a tool for generating synthetic traffic flow data with subject matter specificity. The generation system is designed to be integrated into different systems, which will allow different ITS vendors to use it. This service fulfils the scalability requirements and is close to real data. The study proposes a scalable architecture of intelligent transport subsystem that fulfils the requirements of scalability and fault tolerance. As part of this work, a testbed is assembled and the proposed architecture is tested through the developed service of synthetic traffic flow state data generation.

RPL is a routing protocol for low power and lossy networks. It is an extensible proactive IPv6 distance vector protocol, which is a brand new proposed standard for including low power and lossy networks in the global realm of IPv6 networks. RPL has been implemented and tested in wireless sensor networks (WSNs) simulations with up to a thousand nodes. Recently, some of the research process has studied the operation of the protocol experimentally in small networks, which is not enough to illustrate the whole features of RPL. So, it is important to implement real large scenarios. In this paper, we present an experimental performance evaluation of RPL for different network settings to understand the impact of the protocol attributes on the network behavior, namely in terms of convergence time, energy, packet loss and packet delay.

This paper presents the results of the characterization of 19 industrial customerpsilas measurements of harmonics in their distribution system of medium voltage level, jointly with a software application to carry out a probabilistic evaluation of levels of distortion of currents and harmonic voltages on their systems. The developed program is based on Monte Carlopsilas simulation technique and it was tested on a distribution feeder circuit. The most important results of this work are the following: A dominant behavior of the harmonic order 3, 5, 7 and 11 was found both in current and in voltage; correlation factors for the magnitudes of the harmonic currents for the three phases is nearly 1; there is a significant presence of multimodal probability density functions in all the analyzed data; with more realistic simulation scenarios the average of total harmonic distortion THD v and THD i indicators are reduced, but standard deviations are increased.

Fingerprinting of various physical and logical devices has been proposed for uniquely identifying users or devices of mainstream IT systems such as PCs, laptops, and smart phones. However, the application of such techniques in Industrial Control Systems (ICS) is less explored for reasons such as a lack of direct access to such systems and the cost of faithfully reproducing realistic threat scenarios. This work addresses the feasibility of using fingerprinting techniques in the context of realistic ICS related to water treatment and distribution systems. A model-free sensor fingerprinting scheme ( NoiSense ) and a model-based sensor fingerprinting scheme ( NoisePrint ) are proposed. Using extensive experimentation with sensors, it is shown that noise patterns due to microscopic imperfections in hardware manufacturing can uniquely identify sensors with accuracy as high as 97%. The proposed technique can be used to detect physical attacks, such as the replacement of legitimate sensors by faulty or manipulated sensors. For NoisePrint , a combined fingerprint for sensor and process noise is created. The difference (called residual), between expected and observed values, i.e., noise, is used to derive a model of the system. It was found that in steady state the residual vector is a function of process and sensor noise. Data from experiments reveals that a multitude of sensors can be uniquely identified with a minimum accuracy of 90% based on NoisePrint . Also proposed is a novel challenge-response protocol that exposes more powerful cyber-attacks, including replay attacks.

Network intrusion detection (NID) models increasingly rely on learning traffic microstructures that consist of pattern sequences in features such as interarrival time, size, or packet flags. We argue that precise and reproducible control over traffic microstructures is crucial to understand and improve NID-model behaviour. We demonstrate that probing a traffic classifier with appropriately generated microstructures reveals links between misclassifications and traffic characteristics, and correspondingly lets us improve the false positive rate by more than \(500\%\). We examine how specific factors such as network congestion, load, conducted activity, or protocol implementation impact traffic microstructures, and how well their influence can be isolated in a controlled and near-deterministic traffic generation process. We then introduce DetGen, a traffic generation tool that provides precise microstructure control, and demonstrate how to generate traffic suitable to probe pre-trained NID-models.

Among the several components of distribution systems, protection devices present a fundamental importance, since they aim at keeping the physical integrity not only of the system equipment, but also of the electricians and the population in general. The correct application of the protection devices demand an elevated amount of time, being extremely laborious due to the great number of devices, besides the very dynamic behavior of distribution networks and the need for constant expansion. This article presents a computational tool developed with the objective of automatically determining the adjustments of all protection devices in the distribution networks so as to obtain the best technical application, optimizing its performance and expediting the making of protection studies.

The efficiency and quality of the manufacturing industry are greatly influenced by production scheduling, which makes it a crucial aspect. A well-designed production scheduling scheme can significantly enhance manufacturing efficiency and reduce enterprise costs. This paper presents a tailored optimization model designed to address a more complex production scheduling problem that incorporates parallel machines and preventive maintenance. The proposed solutions aim to achieve a balance between job sequence and machine reliability, considering the minimum maintenance cost rate for determining maintenance cycles of deteriorating machines in real manufacturing scenarios. Furthermore, the objective of minimizing the maximum completion time guides machine assignment and job sequence based on maintenance constraints. The innovation lies in the introduction of a greedy algorithm that utilizes a water injection model to address this NP-hard integrated problem. A pre-distribution model is constructed using the water injection model, and its solution is utilized as input for constructing the production scheduling model, which aids in determining machine assignment and job sequence. This algorithm demonstrates remarkable effectiveness and efficiency, enabling the achievement of an optimal solution. A numerical example is presented to illustrate the computational process, accompanied by an extensive discussion of the results showcasing improved performance. Furthermore, the optimization model developed in this paper can be adapted to tackle the production scheduling problem with modifications tailored for parallel machines.