Achieving better Resource Utilization by Implementing a High-Performance Intelligent Framework in a Distributed Environment

Scientific collaboration is critical in high performance computational (HPC) environments. Computer security incidents, however, are a constant threat in today's interconnected computational environments. Many sites struggle with finding a balance between the needs of users and the threat from unauthorized usage or access. Understanding the constantly changing battlefield is critical for effective and optimal deployment of computer security resources.This tutorial will provide an underlying knowledge of the current state of computer security, its effects on HPC environments and mitigation strategies. Topics we will address include:1.HPC Security Fundamentals2.HPC Security Protection Fundamentals3.Intrusion Detection Using Bro4.The Changing Computer Protection Environment5.Cross-site Implications of Collaborative Environments6.Keeping One Step Ahead7.Risk Assessment, Mitigation and Compliance8.Components of a Good Protection ProgramWe will also show how described techniques are in use at SC06 to protect SCinet.

The efficient implementation of a diffraction tomography method for medical imaging is addressed within the framework of High Performance Computing (HPC) environment. A non-linear optimization method for the solution of the inverse scattering problem is implemented on a shared memory model computer. Linear speed-up and significant reduction in the total execution time is achieved when the program is executed in parallel, enabling the feasibility of the method for realistic medical imaging applications.

Purpose The present paper aims to address the description of a numerical optimization procedure, based on mesh morphing, and its application for the improvement of the aerodynamic performance of an industrial glider which suffers of a large separation occurring in the wing–fuselage junction region at high incidence angles. Design/methodology/approach Shape variations were applied to the baseline configuration through a mesh morphing technique founded on the mathematical framework of radial basis functions (RBF). The aerodynamic solutions were obtained coupling an RANS code with the mesh morphing tool RBF Morph™. Two shape modifiers were set up to generate a parametric numerical model. An optimization procedure, based on a design of experiment sampling, was set up implementing the fully automated workflow within a high performance computing (HPC) environment. The optimal candidates maximizing the aerodynamic efficiency were identified by means of a cubic RBF response surface approach. Findings The separation was significantly reduced, modifying the local geometry of fuselage and fairing and maintaining the wing aerofoil unchanged. A relevant aerodynamic efficiency improvement was finally gained. Practical implications The developed procedure proved to be a very powerful and efficient tool in facing aerodynamic design problems. However, it might be computationally very expensive if a large number of design variables are adopted and, in those cases, the method can be suitably used only within the HPC environment. Originality/value Such an optimization study is part of an explorative set of analyses that focused on better addressing the numerical strategies to be used in the development of the EU FP7 Project RBF4AERO.

The lightweight nature, application portability, and deployment flexibility of containers is driving their widespread adoption in cloud solutions. Data analysis and deep learning (DL)/machine learning (ML) applications have especially benefited from containerization. As such data analysis is adopted in high performance computing (HPC), the need for container support in HPC has become paramount. However, containers face crucial performance and I/O challenges in HPC. One obstacle is that while there have been HPC containers, such solutions have not been thoroughly investigated, especially from the aspect of their impact on the crucial HPC I/O throughput. To this end, this paper provides a first-of-its-kind empirical analysis of state-of-the-art representative container solutions (Docker, Podman, Singularity, and Charliecloud) in HPC environments. We also explore how containers interact with an HPC parallel file system like Lustre. We present the design of an analysis framework that is deployed on all nodes in an HPC environment, and captures CPU, memory, network, and file I/O statistics from the nodes and the storage system. We are able to garner key insights from our analysis, e.g., Charliecloud outperforms other container solutions in terms of container start-up time, while Singularity and Charliecloud are equivalent in I/O throughput. But this comes at a cost, as Charliecloud invokes the most metadata and I/O operations on the underlying Lustre file system. By identifying such trade-offs and optimization opportunities, we can enhance HPC containers performance and the ML/DL applications that increasingly rely on them.

SummaryThe wide range of RNA-seq applications and their high-computational needs require the development of pipelines orchestrating the entire workflow and optimizing usage of available computational resources. We present aRNApipe, a project-oriented pipeline for processing of RNA-seq data in high-performance cluster environments. aRNApipe is highly modular and can be easily migrated to any high-performance computing (HPC) environment. The current applications included in aRNApipe combine the essential RNA-seq primary analyses, including quality control metrics, transcript alignment, count generation, transcript fusion identification, alternative splicing and sequence variant calling. aRNApipe is project-oriented and dynamic so users can easily update analyses to include or exclude samples or enable additional processing modules. Workflow parameters are easily set using a single configuration file that provides centralized tracking of all analytical processes. Finally, aRNApipe incorporates interactive web reports for sample tracking and a tool for managing the genome assemblies available to perform an analysis.Availability and documentationhttps://github.com/HudsonAlpha/aRNAPipe; DOI: 10.5281/zenodo.202950Supplementary informationSupplementary data are available at Bioinformatics online.

Cloud computing and big data: Technologies and applications

Abstract. Most Earth system models (ESMs) are running under different high-performance computing (HPC) environments. This has several advantages, from allowing different groups to work with the same tool in parallel to leveraging the burden of ensemble climate simulations, but it also offers alternative solutions in the case of shutdown (expected or not) of any of the environments. However, for obvious scientific reasons, it is critical to ensure that ESMs provide identical results under changes in computing environment. While strict bit-for-bit reproducibility is not always guaranteed with ESMs, it is desirable that results obtained under one computing environment are at least statistically indistinguishable from those obtained under another environment, which we term a “replicability” condition following the metrology nomenclature. Here, we develop a protocol to assess the replicability of the EC-Earth ESM. Using two versions of EC-Earth, we present one case of non-replicability and one case of replicability. The non-replicable case occurs with the older version of the model and likely finds its origin in the treatment of river runoff along Antarctic coasts. By contrast, the more recent version of the model provides replicable results. The methodology presented here has been adopted as a standard test by the EC-Earth consortium (27 institutions in Europe) to evaluate the replicability of any new model version across platforms, including for CMIP6 experiments. To a larger extent, it can be used to assess whether other ESMs can safely be ported from one HPC environment to another for studying climate-related questions. Our results and experience with this work suggest that the default assumption should be that ESMs are not replicable under changes in the HPC environment, until proven otherwise.

98P - Standardized and compliant bio-informatics pipeline for neoepitope analysis in oncology clinical trials

Task scheduling is one of the very crucial facets of cloud computing. The task scheduling method must assign jobs to virtual machines. In cloud computing, task scheduling includes a frontal influence on a system's resource utilization and operational costs. Diverse meta-heuristic algorithms, in addition to their modifications, have been developed to improve the efficiency of task executions in the cloud. In this paper, a multiobjective optimization model is applied using the metaheuristics cuckoo search optimization algorithm (MCSO) to enhance the performance of a cloud system with limited computing resources while minimizing the time and cost. Finally, we analyze the performance of the proposed MCSO with the existing methods, such as Bee Life Algorithm (BLA), A Time–Cost aware Scheduling (TCaS) algorithm, Modified Particle Swarm Optimization (MPSO), and Round Robin (RR), for the evaluation metrics makespan and cost. Based on the outcomes of the experiments, it can be inferred that the proposed MCSO provides essential schedule jobs with the shortest makespan and average cost. Index Terms— Task Scheduling, Multiobjective, Cloud Computing, Cost, Makespan.

Future computer systems will integrate tens of multithreaded processor cores on a single chip die, resulting in hundreds of concurrent program threads sharing system resources. These designs will be the cornerstone of improving throughput in high-performance computing and server environments. However, to date, appropriate systems software (operating system, run-time system, and compiler) technologies for these emerging machines have not been adequately explored. Future processors will require sophisticated hardware monitoring units to continuously feed back resource utilization information to allow the operating system to make optimal thread co-scheduling decisions and also to software that continuously optimizes the program itselfNevertheless, in order to continually and automatically adapt systems resources to program behaviors and application needs, specific run-time information must be collected to adequately enable dynamic code optimization and operating system scheduling. Generally, run-time optimization is limited by the time required to collect profiles, the time required to perform optimization, and the inherent benefits of any optimization or decisions. Initial techniques for effectively utilizing run-time information for dynamic optimization and informed thread scheduling in future multithreaded architectures are presented

This chapter gives a comprehensive overview of the current status of accounting and billing for up-todate computing environments. Accounting is the key for the management of information system resources. At this stage of evolution of accounting systems it is adequate not to separate computing environments into High Performance Computing and Grid Computing environments for allowing a “holistic” view showing the different approaches and the state of the art for integrated accounting and billing in distributed computing environments. Requirements resulting from a public survey within all communities of the German Grid infrastructure, as well as from computing centres and resource providers of High Performance Computing resources like HLRN, and ZIVGrid, within the German e-Science framework, have been considered as well as requirements resulting from various information systems and the virtualisation of organisations and resources. Additionally, conceptual, technical, economical, and legal questions also had to be taken into consideration. After the requirements have been consolidated and implementations have been done over one year ago, now the overall results and conclusions are presented in the following sections showing a case study based on the GISIG framework and the Grid- GIS framework. The focus is on how an integrated architecture can be built and used in heterogeneous environments. A prototypical implementation is outlined that is able to manage and visualise relevant accounting and billing information based on suitable monitoring data in a virtual organisation (VO) specific way regarding basic business, economic, and security issues.

In cloud computing, optimizing task scheduling is crucial for improving overall system performance and resource utilization. To minimize cloud service costs and prevent resource wastage, advanced techniques must be employed to efficiently allocate cloud resources for executing tasks. This research presents a novel multi-objective task scheduling method, BSSA, which combines the Backtracking Search Optimization Algorithm (BSA) and the Sparrow Search Algorithm (SSA). BSA enhances SSA’s convergence accuracy and global optimization ability in later iterations, improving task scheduling results. The proposed BSSA is evaluated and compared against traditional SSA and other algorithms using a set of 8 benchmark test functions. Moreover, BSSA is tested for task scheduling in cloud environments and compared with various metaheuristic scheduling algorithms. Experimental results demonstrate the superiority of the proposed BSSA, validating its effectiveness and efficiency in cloud task scheduling.

In Cloud Computing (CC) environment, load balancing refers to the process of optimizing resources of virtual machines. Load balancing in the CC environment is one of the analytical approaches utilized to ensure indistinguishable workload distribution and effective utilization of resources. This is because only by ensuring effective balance of dynamic workload results in higher user satisfaction and optimal allocation of resource, therefore improve cloud application performance. Moreover, a paramount objective of load balancing is task scheduling because surges in the number of clients utilizing cloud lead to inappropriate job scheduling. Hence, issues encircling task scheduling has to be addressed. In this work a method called, Ford Fulkerson and Newey West Regression-based Dynamic Load Balancing (FF-NWRDLB) in CC environment is proposed. The FF-NWRDLB method is split into two sections, namely, task scheduling and dynamic load balancing. First, Ford Fulkerson-based Task Scheduling is applied to the cloud user requested tasks obtained from Personal Cloud Dataset. Here, employing Ford Fulkerson function based on the flow of tasks, energy-efficient task scheduling is ensured. The execution of asymmetrical scientific applications can be smoothly influenced by an unbalanced workload distribution between computing resources. In this context load balancing signifies as one of the most significant solution to enhance utilization of resources. However, selecting the best accomplishing load balancing technique is not an insignificant piece of work. For example, selecting a load balancing model does not work in circumstances with dynamic behavior. In this context, a machine learning technique called, Newey West Regression-based dynamic load balancer is designed to balance the load in a dynamic manner at run time, therefore ensuring accurate data communication. The FF-NWRDLB method has been compared to recent algorithms that use the markov optimization and the prediction scheme to achieve load balancing. Our experimental results show that our proposed FF-NWRDLB method outperforms other state of the art schemes in terms of energy consumption, throughput, delay, bandwidth and task scheduling efficiency in CC environment.

Scientific workflows are models composed of activities, data and dependencies whose objective is to represent a computer simulation. Workflows are managed by Scientific Workflow Management System (SWfMS). Such workflows commonly demand for many computational resources once their executions may involve a number of different programs processing a huge volume of data. Thus, the use of High Performance Computing (HPC) environments allied to parallelization techniques provides the support for the execution of such experiments. Some resources provided by clouds can be used to build HPC environments. Although clouds offer advantages such as elasticity and availability, failures are a reality rather than a possibility. Thus, SWfMS must be fault-tolerant. There are several types of fault tolerance techniques used in SWfMS such as checkpoint-restart and replication, but which fault tolerance technique best fits with a specific workflow? This work aims at analyzing several fault tolerance techniques in SWfMSs and recommending the suitable one for the user’s workflow using machine learning techniques and provenance data, thus improving resiliency.

Arc4nix: A cross-platform geospatial analytical library for cluster and cloud computing