A browser-based platform for storage, visualization, and analysis of large-scale 3D images in HPC environments

LiDAR is an active remote sensing technique with a unique capability to capture three-dimensional information of the earth’s surface even in heavily vegetated areas, and it is proven to be useful in many research applications. Although it is becoming the remote sensing platform of choice for planning and natural resource agencies that require three-dimensional information, the enormous data that are generated and the lack of available software analysis packages make LiDAR still unavailable to a typical user of spatial data. LiDARHub is a free and open source platform for web-based management, visualization and analysis of LiDAR data that enables development of online tools for LiDAR data processing in a web browser. The framework provides a foundation to develop online tools for LiDAR data processing and tools can be shared. The framework is also flexible so that the developed tools can be easily ported to High Performance Computing (HPC) environments that speed up the computationally extensive LiDAR data processing. Two example LiDARHub tools are presented as case studies to demonstrate potential software development scenarios. The developed tools provide easy to use user interface and hide complex computation so that users can take advantage of the LiDAR technology with only a web browser. The LiDARHub allows not only the sharing of large volume of LiDAR data but also developing online LiDAR processing platform for a large audience.

Scientific collaboration is critical in high performance computational (HPC) environments. Computer security incidents, however, are a constant threat in today's interconnected computational environments. Many sites struggle with finding a balance between the needs of users and the threat from unauthorized usage or access. Understanding the constantly changing battlefield is critical for effective and optimal deployment of computer security resources.This tutorial will provide an underlying knowledge of the current state of computer security, its effects on HPC environments and mitigation strategies. Topics we will address include:1.HPC Security Fundamentals2.HPC Security Protection Fundamentals3.Intrusion Detection Using Bro4.The Changing Computer Protection Environment5.Cross-site Implications of Collaborative Environments6.Keeping One Step Ahead7.Risk Assessment, Mitigation and Compliance8.Components of a Good Protection ProgramWe will also show how described techniques are in use at SC06 to protect SCinet.

<p>The complexity of the climate system calls for a combined approach of different knowledge areas. For that, increasingly larger projects need a coordinate effort that fosters an active collaboration between members. On the other hand, although the continuous improvement of numerical models and larger observational data availability provides researchers with a growing amount of data to analyze, the need for greater resources to host, access, and evaluate them efficiently through High Performance Computing (HPC) infrastructures is growing more than ever. Finally, the thriving emphasis on FAIR data principles [1] and the easy reproducibility of evaluation workflows also requires a framework that facilitates these tasks. Freva (Free Evaluation System Framework [2, 3]) is an efficient solution to handle customizable evaluation systems of large research projects, institutes or universities in the Earth system community [4-6] over the HPC environment and in a centralized manner.</p><p> </p><p>Freva is a scientific software infrastructure for standardized data and analysis tools (plugins) that provides all its available features both in a shell and web environment. Written in python, is equipped with a standardized model database, an application-programming interface (API) and a history of evaluations, among others:</p><ul><li>An implemented metadata system in SOLR with its own search tool allows scientists and their plugins to retrieve the required information from a centralized database. The databrowser interface satisfies the international standards provided by the Earth System Grid Federation (ESGF, e.g. [7]).</li> <li>An API allows scientific developers to connect their plugins with the evaluation system independently of the programming language. The connected plugins are able to access from and integrate their results back to the database, allowing for a concatenation of plugins as well. This ecosystem increases the number of scientists involved in the studies, boosting the interchange of results and ideas. It also fosters an active collaboration between plugin developers.</li> <li>The history and configuration sub-system stores every analysis performed with Freva in a MySQL database. Analysis configurations and results can be searched and shared among the scientists, offering transparency and reproducibility, and saving CPU hours, I/O, disk space and time.</li> </ul><p>Freva efficiently frames the interaction between different technologies thus improving the Earth system modeling science.</p><p> </p><p>This framework has undergone major refactoring and restructuring of the core that will also be discussed. Among others:</p><ul><li>Major core Python update (2.7 to 3.9).</li> <li>Easier deployment and containerization of the framework via Docker.</li> <li>More secure system configuration via Vault integration.</li> <li>Direct Freva function calls via python client (e.g. for jupyter notebooks).</li> <li>Improvements in the dataset incorporation.</li> </ul><p><strong> </strong></p><p><strong>

The lightweight nature, application portability, and deployment flexibility of containers is driving their widespread adoption in cloud solutions. Data analysis and deep learning (DL)/machine learning (ML) applications have especially benefited from containerization. As such data analysis is adopted in high performance computing (HPC), the need for container support in HPC has become paramount. However, containers face crucial performance and I/O challenges in HPC. One obstacle is that while there have been HPC containers, such solutions have not been thoroughly investigated, especially from the aspect of their impact on the crucial HPC I/O throughput. To this end, this paper provides a first-of-its-kind empirical analysis of state-of-the-art representative container solutions (Docker, Podman, Singularity, and Charliecloud) in HPC environments. We also explore how containers interact with an HPC parallel file system like Lustre. We present the design of an analysis framework that is deployed on all nodes in an HPC environment, and captures CPU, memory, network, and file I/O statistics from the nodes and the storage system. We are able to garner key insights from our analysis, e.g., Charliecloud outperforms other container solutions in terms of container start-up time, while Singularity and Charliecloud are equivalent in I/O throughput. But this comes at a cost, as Charliecloud invokes the most metadata and I/O operations on the underlying Lustre file system. By identifying such trade-offs and optimization opportunities, we can enhance HPC containers performance and the ML/DL applications that increasingly rely on them.

Purpose The present paper aims to address the description of a numerical optimization procedure, based on mesh morphing, and its application for the improvement of the aerodynamic performance of an industrial glider which suffers of a large separation occurring in the wing–fuselage junction region at high incidence angles. Design/methodology/approach Shape variations were applied to the baseline configuration through a mesh morphing technique founded on the mathematical framework of radial basis functions (RBF). The aerodynamic solutions were obtained coupling an RANS code with the mesh morphing tool RBF Morph™. Two shape modifiers were set up to generate a parametric numerical model. An optimization procedure, based on a design of experiment sampling, was set up implementing the fully automated workflow within a high performance computing (HPC) environment. The optimal candidates maximizing the aerodynamic efficiency were identified by means of a cubic RBF response surface approach. Findings The separation was significantly reduced, modifying the local geometry of fuselage and fairing and maintaining the wing aerofoil unchanged. A relevant aerodynamic efficiency improvement was finally gained. Practical implications The developed procedure proved to be a very powerful and efficient tool in facing aerodynamic design problems. However, it might be computationally very expensive if a large number of design variables are adopted and, in those cases, the method can be suitably used only within the HPC environment. Originality/value Such an optimization study is part of an explorative set of analyses that focused on better addressing the numerical strategies to be used in the development of the EU FP7 Project RBF4AERO.

The efficient implementation of a diffraction tomography method for medical imaging is addressed within the framework of High Performance Computing (HPC) environment. A non-linear optimization method for the solution of the inverse scattering problem is implemented on a shared memory model computer. Linear speed-up and significant reduction in the total execution time is achieved when the program is executed in parallel, enabling the feasibility of the method for realistic medical imaging applications.

BackgroundRecent years have seen a substantial increase in RNA-seq data production, with this technique becoming the primary approach for gene expression studies across a wide range of non-model organisms. The majority of these organisms lack a well-annotated reference genome to serve as a basis for studying differentially expressed genes (DEGs). As an alternative cost-effective protocol to using a reference genome, the assembly of RNA-seq raw reads is performed to produce what is referred to as a ‘de novo transcriptome,’ serving as a reference for subsequent DEGs’ analysis. This assembly step for conventional DEGs analysis pipelines for non-model organisms is a computationally expensive task. Furthermore, the complexity of the de novo transcriptome assembly workflows poses a challenge for researchers in implementing best-practice techniques and the most recent software versions, particularly when applied to various organisms of interest.ResultsTo address computational challenges in transcriptomic analyses of non-model organisms, we present HPC-T-Assembly, a tool for de novo transcriptome assembly from RNA-seq data on high-performance computing (HPC) infrastructures. It is designed for straightforward setup via a Web-oriented interface, allowing analysis configuration for several species. Once configuration data is provided, the entire parallel computing software for assembly is automatically generated and can be launched on a supercomputer with a simple command line. Intermediate and final outputs of the assembly pipeline include additional post-processing steps, such as assembly quality control, ORF prediction, and transcript count matrix construction.ConclusionHPC-T-Assembly allows users, through a user-friendly Web-oriented interface, to configure a run for simultaneous assemblies of RNA-seq data from multiple species. The parallel pipeline, launched on HPC infrastructures, significantly reduces computational load and execution times, enabling large-scale transcriptomic and meta-transcriptomics analysis projects.

98P - Standardized and compliant bio-informatics pipeline for neoepitope analysis in oncology clinical trials

SummaryThe wide range of RNA-seq applications and their high-computational needs require the development of pipelines orchestrating the entire workflow and optimizing usage of available computational resources. We present aRNApipe, a project-oriented pipeline for processing of RNA-seq data in high-performance cluster environments. aRNApipe is highly modular and can be easily migrated to any high-performance computing (HPC) environment. The current applications included in aRNApipe combine the essential RNA-seq primary analyses, including quality control metrics, transcript alignment, count generation, transcript fusion identification, alternative splicing and sequence variant calling. aRNApipe is project-oriented and dynamic so users can easily update analyses to include or exclude samples or enable additional processing modules. Workflow parameters are easily set using a single configuration file that provides centralized tracking of all analytical processes. Finally, aRNApipe incorporates interactive web reports for sample tracking and a tool for managing the genome assemblies available to perform an analysis.Availability and documentationhttps://github.com/HudsonAlpha/aRNAPipe; DOI: 10.5281/zenodo.202950Supplementary informationSupplementary data are available at Bioinformatics online.

Building and sustaining High-Performance Computing (HPC) infrastructure for bioinformatics research in resource-limited settings presents significant technical, financial and operational challenges. Institutions in low-and middle-income regions often face constraints such as limited technical expertise, unstable infrastructure and restricted funding which can hinder the deployment of large-scale computational platforms necessary for modern genomics and bioinformatics analyses. We present a scalable and modular HPC framework developed at the Uganda Virus Research Institute (UVRI) to support large-scale genomics and other omics data analyses in resource-limited settings. The framework integrates open-source HPC management tools, infrastructure automation, and reproducible configuration management to enable reliable deployment and maintenance. Optimized storage and networking configurations combined with a phased capacity-building strategy support high-throughput genomic workflows while strengthening local technical expertise. From our implementation experience, we derive ten practical design and operational rules that provide a transferable methodology for establishing and sustaining in-house HPC infrastructure. These rules emphasize strategic investment in human capacity, structured planning, leveraging collaborations, adoption of open-source technologies and service management practices to improve operational resilience and long-term sustainability. The design principles, automation strategies and implementation guidelines described in this work are applicable to institutions seeking to establish sustainable HPC resources for bioinformatics research in resource-constrained environments.

Deep learning integral imaging for three-dimensional visualization, object detection, and segmentation

Tomographic imaging is an important disease diagnosis method. Specific image workstation is usually needed to process and visualize 3D medical image gathered from tomographic imaging equipments, because this process is computation intensive. Specific image workstation is large and expensive, which limits size and price of medical tomographic imaging system. To break these limits, this paper presents an embedded 3D medical image processing and visualization platform based on common dual-core processor. The proposed system parallelizes 3D medical image processing and visualization computing process to meet system real-time requirements. Main contributions of this paper includes analysis of core 3D medical data processing and visualization algorithm, program parallelization method analysis and implementation, system performance measurement and analysis. Program analysis results show that the proposed system fully utilizes hardware resource, and meets low-end and space-efficient medical imaging system requirements.

Cloud computing has become an important driver for delivering infrastructure as a service (IaaS) to users with on-demand requests for customized environments and sophisticated software stacks. Within the FutureGrid (FG) project, we offer different IaaS frameworks as well as high performance computing infrastructures by allowing users to explore them as part of the FG testbed. To ease the use of these infrastructures, as part of performance experiments, we have designed an image management framework, which allows us to create user defined software stacks based on abstract image management and uniform image registration. Consequently, users can create their own customized environments very easily. The complex processes of the underlying infrastructures are managed by our sophisticated software tools and services. Besides being able to manage images for IaaS frameworks, we also allow the registration and deployment of images onto bare-metal by the user. This level of functionality is typically not offered in a HPC (high performance computing) infrastructure. However, our approach provides users with the ability to create their own environments changing the paradigm of administrator-controlled dynamic provisioning to user-controlled dynamic provisioning, which we also call raining. Thus, users obtain access to a testbed with the ability to manage state-of-the-art software stacks that would otherwise not be supported in typical compute centers. Security is also considered by vetting images before they are registered in a infrastructure. In this paper, we present the design of our image management framework and evaluate two of its major components. This includes the image creation and image registration. Our design and implementation can support the current FG user community interested in such capabilities.

To improve the estimation of the perceived pitch in a single-sided deaf cochlear implant (CI) listener by using accurate 3-dimensional (3D) image analysis of the cochlear electrode positions together with the predicted tonotopical function for humans. An SSD CI user underwent a Cone-Beam computed tomography (CBCT) scan. Electrode contacts were marked in 3D space in relation to the nearest point on the cochlear lateral wall. Distance to the base of the lateral wall was calculated and plotted against the place-pitch function for humans. An adaptive procedure was used to elicit the perceived pitch of electrically evoked stimulation by matching it with a contralateral acoustic pitch. The electrically evoked pitch percept matched well with the calculated frequency. The median mismatch in octaves was 0.12 for our method in comparison to 0.69 using the conventional Stenvers view. A method of improved image analysis is described that can be used to predict the pitch percept on corresponding cochlear electrode positions. This method shows the potential of 3D imaging in CI fitting optimization.

Modern High Performance Computing (HPC) applications, such as Earth science simulations, produce large amounts of data due to the surging of computing power, while big data applications have become more compute-intensive due to increasingly sophisticated analysis algorithms. The needs of both HPC and big data technologies for advanced HPC and big data applications create a demand for integrated system support. In this study, we introduce Scientific Data Processing (SciDP) to support both HPC and big data applications via integrated scientific data processing. SciDP can directly process scientific data stored on a Parallel File System (PFS), which is typically deployed in an HPC environment, in a big data programming environment running atop Hadoop Distributed File System (HDFS). SciDP seamlessly integrates PFS, HDFS, and the widely-used R data analysis system to support highly efficient processing of scientific data. It utilizes the merits of both PFS and HDFS for fast data transfer, overlaps computing with data accessing, and integrates R into the data transfer process. Experimental results show that SciDP accelerates analysis and visualization of a production NASA Center for Climate Simulation (NCCS) climate and weather application by 6x to 8x when compared to existing solutions.