HPC Annual Report 2017

This project addresses three important gaps in existing evaluated nuclear data libraries that represent a significant hindrance against highly advanced modeling and simulation capabilities for the Advanced Fuel Cycle Initiative (AFCI). This project will: Develop advanced theoretical tools to compute prompt fission neutrons and gamma-ray characteristics well beyond average spectra and multiplicity, and produce new evaluated files of U and Pu isotopes, along with some minor actinides; Perform state-of-the-art fission cross-section modeling and calculations using global and microscopic model input parameters, leading to truly predictive fission cross-sections capabilities. Consistent calculations for a suite of Pu isotopes will be performed; Implement innovative data assimilation tools, which will reflect the nuclear data evaluation process much more accurately, and lead to a new generation of uncertainty quantification files. New covariance matrices will be obtained for Pu isotopes and compared to existing ones. The deployment of a fleet of safe and efficient advanced reactors that minimize radiotoxic waste and are proliferation-resistant is a clear and ambitious goal of AFCI. While in the past the design, construction and operation of a reactor were supported through empirical trials, this new phase in nuclear energy production is expected to rely heavily on advanced modeling and simulation capabilities. To be truly successful, a program for advanced simulations of innovative reactors will have to develop advanced multi-physics capabilities, to be run on massively parallel super- computers, and to incorporate adequate and precise underlying physics. And all these areas have to be developed simultaneously to achieve those ambitious goals. Of particular interest are reliable fission cross-section uncertainty estimates (including important correlations) and evaluations of prompt fission neutrons and gamma-ray spectra and uncertainties.

Microreactors are small mobile transportable nuclear reactors with thermal power less than 20 MWt that are designed to meet the needs of remote areas, military installations, emergency operations, and disaster relief zones. These reactor are concepts that are intended to be factory-manufacturable, easily transportable, and will allow for semi- or full-autonomous operation. To assist with development of microreactors, modeling and experimentation need to be performed with verification and validation to gain confidence in the models and tools. The objective of this study is apply advanced modeling and simulation capabilities being developed by the DOE Office of Nuclear Energy Nuclear Energy Advanced Modeling and Simulation (NEAMS) program to perform integrated modeling and simulation capability for multiphysics simulation of the microreactor concepts. This activity will provide an opportunity to exercise the codes to provide both gain experience and to provide feedback to the developers.

The pebble bed reactor is a unique reactor design due to its capacity for continuous multipass circulation of the fuel elements, without causing interruption to reactor operation, with the assistance of the burnup measurement system. Such a system necessarily requires an accurate knowledge of the burnup of each fuel pebble upon ejection from the core so as to inform the reloading decision and to ensure that no pebble exceeds the regulated discharge burnup limit at any point following reinsertion into the reactor core. In this work, we conceptualize, develop, and demonstrate a machine learning–based fuel burnup prediction framework leveraging advanced modeling and simulation capabilities. At its core, machine learning regression models are learned from simulated data to establish the correlation among the irradiated fuel composition (hence burnup), the gamma leakage spectrum, and the gamma spectroscopy results. Sensitivity analysis is conducted to quantify the impact of unknown design parameters, such as fuel enrichment, and irradiation environment, including power density, temperature, and neighboring materials, on the prediction accuracy of various supervised regression algorithms. The effects of a short cooldown period on machine learning prediction accuracy are also investigated. A test data set is used to validate that the data generation methodology proposed in this work successfully results in a machine learning model capable of interpolating its prediction of burnup onto a much wider range of irradiation conditions than were explicitly represented in the training database. The inclusion of a cooldown period of just 2 h leads to a prediction root-mean-square error of <5 MWd/kgU when the fuel enrichment is known and <9 MWd/kgU otherwise.

The Horizon Europe project &amp;#8220;Biodiversity Digital Twin for Advanced Modelling, Simulation and Prediction Capabilities&amp;#8221; (biodt.eu) combines state-of-the-art supercomputing resources with FAIR data practices to address biodiversity grand challenges such as the loss of diversity at genetic, species and ecosystem levels driven by factors such as anthropogenic climate change, intensified land use and the spreading of alien invasive species.Key tools for the integration and assessment of such information in BioDT are so-called prototype Digital Twins (pDTs) which provide digital replicas of particular phenomena of the biosphere. The data produced by pDTs and the computational workflows themselves are annotated with rich machine-interpretable metadata, notably schema.org and its Bioschemas and Croissant (high-level metadata format for machine learning datasets) extensions to enable discovery and interaction with the data for humans and machines.&amp;#160;The crop wild relatives digital twin developed in BioDT for the identification of novel genetic resources in crop wild relatives (CWR) aims to support the development of mitigation strategies against food-related crises associated with climate change (doi:10.3897/rio.10.e125192). Underutilised crops and CWR are a valuable source of untapped genetic diversity for implementing such strategies because their greater genetic diversity in relation to crops makes adaptation to droughts, cold waves, heavy precipitation or other weather-induced extreme events possible. They are therefore of particular importance to guarantee food security.&amp;#160;&amp;#160;We developed a pilot study to migrate and functionally integrate the pDT CWR in the Destination Earth Data Lake (DEDL) to increase its availability and, more importantly, use both data provided in the data space and the near data processing features of DEDL to improve models and prediction. The framework makes use of web-based technologies such as RO-Crates and FAIR Signposting to facilitate reuse and repurposing of the data within and across different data spaces as well as orchestrated interplay of multiple digital twins (involving the Climate Change Adaptation Digital Twin).&amp;#160;Building on this blueprint, we will showcase in this presentation the deployment of FAIR agrobiodiversity workflows in DEDL&amp;#8217;s compute service Islet and the subsequent publication of results in a lightweight service framework (figure).

We present new visualization tools for the recently developed High Performance Computing (HPC) capability for the MICHELLE code. New HPC support has been implemented in the MICHELLE code in support of rapid device design and optimization. The adoption of HPC tools and the porting of the code to HPC platforms allows for high -resolution, large-scale models and simulations of multiple beam devices, such as MBK guns and MB-IOTs. Visual analysis and postprocessing of simulation results from such large-scale models is a computational challenge that requires advanced scientific visualization tools such as Para View, which was chosen for our program due to its flexibility to plug-in customization. To enhance usability and efficiency for new and existing MICHELLE users, a GUI plug-in to ParaView with MICHELLE-specific tools has been developed using the Qt GUI toolkit and Python. Here we present a description of the Qt plug-in with example problems, and a review of some Para View capabilities. Also presented is real-world benchmarking of HPC visualization performance using Para View's in client-server mode.

Heterogeneous sparse matrix–vector multiplication via compressed sparse row format

: In the face of increasing competition from other market economies, the US needs to ramp up its ability to innovate and High Performance Computing (HPC) must be a key ingredient in America's capacity. The task ahead therefore must include broadening the use of HPC in US industry through collaborations in which more knowledgeable HPC users especially leading government users share their expertise and advanced computing resources with less-experienced and inexperienced private sector companies. We must move beyond today's islands of innovation by making access to HPC much easier and more pervasive. Leveraging these world-class assets to their fullest is critical if we are going to successfully harness America's capacity to drive future economic growth and industrial leadership. In today's high competitive global economy, the country that wants to outcompete must be able to out-compute. This report summarizes an initiative whose objective was to attempt to assure a path for realizing dual-use applications for next generation high performance computing.

Smart grids are a crucial component for enabling the nation’s future energy needs, as part of a modernization effort led by the Department of Energy. Smart grids and smart microgrids are being considered in niche applications, and as part of a comprehensive energy strategy to help manage the nation’s growing energy demands, for critical infrastructures, military installations, small rural communities, and large populations with limited water supplies. As part of a far-reaching strategic initiative, Sandia National Laboratories (SNL) presents herein a unique, three-pronged approach to integrate small modular reactors (SMRs) into microgrids, with the goal of providing economically-competitive, reliable, and secure energy to meet the nation’s needs. SNL’s triad methodology involves an innovative blend of smart microgrid technology, high performance computing (HPC), and advanced manufacturing (AM). In this report, Sandia’s current capabilities in those areas are summarized, as well as paths forward that will enable DOE to achieve its energy goals. In the area of smart grid/microgrid technology, Sandia’s current computational capabilities can model the entire grid, including temporal aspects and cyber security issues. Our tools include system development, integration, testing and evaluation, monitoring, and sustainment.

Molten Salt Reactors display a strong coupling between neutronics and thermal-hydraulics for which various simulation tools have been developed. In this paper we describe TRUST-NK, a neutronics code based on TRUST, CEA’s High Performance Computing (HPC) platform. TRUST-NK provides a multi-group diffusion solver with transport of delayed neutron precursors, for both steady-state and transient simulations. Being based on TRUST allows TRUST-NK to take advantage of the platform’s HPC capabilities and to easily couple to CFD solvers provided by other derivatives of TRUST. After a brief description of the code, we present an example of application to a simplified MSR model.

In order to guaranty performances of complex systems using numerical simulation, CEA is performing advanced data analysis and scientific visualization with open source software using High Performance Computing (HPC) capability. The diversity of the physics to study produces results of growing complexity in terms of large-scale, high dimensional and multivariate data. Moreover, the HPC approach introduces another layer of complexity by allowing computation amongst thousands of remote cores accessed from sites located hundreds of kilometers away from the computing facility. This paper presents how CEA deploys and contributes to open source software to enable production class visualization tools in a high performance computing context. Among several open source projects used at CEA, this presentation will focus on Visit, VTK and Paraview. In the first part we will address specific issues encountered when deploying VisIt and Paraview in a multi-site supercomputing facility for end-users. Several examples will be given on how such tools can be adapted to take advantage of a parallel setting to explore large multi-block dataset or perform remote visualization on material interface reconstructions of billions of cells. Then, the specific challenges faced to deliver Paraview’s Catalyst capabilities to end-users will be discussed. In the second part, we will describe how CEA contributes to open source visualization software and associated software development strategy by emphasizing on two recent development projects. The first is an integrated simulation workbench providing plugins for every step required to achieve numerical simulation independently on a local or a remote computer. Embedded in an Eclipse RCP environment, VTK views allow the users to perform data input using interaction or mesh preview before running the simulation code. Contributions to VTK have been made in order to smoothly integrate these technologies. The second details how recent developments at CEA have helped to visualize and to analyze results from ExaStamp, a parallel molecular dynamics simulation code dealing with molecular systems ranging from a few millions up to a billion atoms. These developments include a GPU intensive rendering method specialized for atoms and specific parallel algorithms to process molecular data sets.

Large-scale scientific computation and all of the disciplines that support and help to validate it have been placed at the focus of Lawrence Livermore National Laboratory (LLNL) by the Advanced Simulation and Computing (ASC) program of the National Nuclear Security Administration (NNSA) and the Scientific Discovery through Advanced Computing (SciDAC) initiative of the Office of Science of the Department of Energy (DOE). The maturation of computational simulation as a tool of scientific and engineering research is underscored in the November 2004 statement of the Secretary of Energy that, ''high performance computing is the backbone of the nation's science and technology enterprise''. LLNL operates several of the world's most powerful computers--including today's single most powerful--and has undertaken some of the largest and most compute-intensive simulations ever performed. Ultrascale simulation has been identified as one of the highest priorities in DOE's facilities planning for the next two decades. However, computers at architectural extremes are notoriously difficult to use efficiently. Furthermore, each successful terascale simulation only points out the need for much better ways of interacting with the resulting avalanche of data. Advances in scientific computing research have, therefore, never been more vital to LLNL's core missions than at present. Computational science is evolving so rapidly along every one of its research fronts that to remain on the leading edge, LLNL must engage researchers at many academic centers of excellence. In Fiscal Year 2004, the Institute for Scientific Computing Research (ISCR) served as one of LLNL's main bridges to the academic community with a program of collaborative subcontracts, visiting faculty, student internships, workshops, and an active seminar series. The ISCR identifies researchers from the academic community for computer science and computational science collaborations with LLNL and hosts them for short- and long-term visits with the aim of encouraging long-term academic research agendas that address LLNL's research priorities. Through such collaborations, ideas and software flow in both directions, and LLNL cultivates its future workforce. The Institute strives to be LLNL's ''eyes and ears'' in the computer and information sciences, keeping the Laboratory aware of and connected to important external advances. It also attempts to be the ''feet and hands'' that carry those advances into the Laboratory and incorporates them into practice. ISCR research participants are integrated into LLNL's Computing and Applied Research (CAR) Department, especially into its Center for Applied Scientific Computing (CASC). In turn, these organizations address computational challenges arising throughout the rest of the Laboratory. Administratively, the ISCR flourishes under LLNL's University Relations Program (URP). Together with the other five institutes of the URP, it navigates a course that allows LLNL to benefit from academic exchanges while preserving national security. While it is difficult to operate an academic-like research enterprise within the context of a national security laboratory, the results declare the challenges well met and worth the continued effort.

&lt;p&gt;The objective of this study is to integrate a continuum-based deformable tire and terrain interaction model into a general-use physics-based simulation environment capable of off-road vehicle mobility analysis and high-performance computing potential. Specifically, the physics-based deformable tire and terrain models which were recently proposed and validated by Yamashita, et al. will be implemented into the structure of the multi-physics simulation engine Chrono. In off-road vehicle mobility analysis, empirical and analytical models have been commonly used for vehicle-terrain interaction. While these models utilize experimental data or terramechanics theories to create quick predictive mobility models, they are unable to capture the highly nonlinear behavior of soft soil deformation, which can lead to inaccurate or unreliable results. In order to resolve these limitations, the use of physics-based numerical approaches have been proposed. These methods make use of finite element and discrete element simulations to describe the interaction between the vehicle and deformable terrain. Continuum-based finite element models transfer tire forces to the terrain and model the deformation with elasto-plastic constitutive models. Discrete element soil uses a large number of small rigid body particles to describe the microscale behavior of granular terrain, with the deformation of the soil represented by the motion and contact of the particles. While these physics-based models offer a more accurate vehicle-terrain interaction model, the solution procedure can become complex and computationally expensive since co-simulation techniques are often used.&lt;/p&gt;&lt;p&gt;To address these issues, the analysis of physics-based full vehicle dynamics simulations utilizing high-fidelity deformable tire and terrain models in a multi-physics engine with high-performance computing capability is desired. To this end, a continuum mechanics based shear deformable laminated composite shell element proposed by Yamashita, et al. was integrated into the flexible body dynamics simulation framework of Chrono. This element was based on the absolute nodal coordinate formulation and is defined by the global position coordinates and the transverse gradient coordinates of its four nodes. Element lockings are eliminated with the incorporation of the enhanced assumed strain (EAS) and assumed natural strain approaches (ANS). The element formulation includes an extension to model laminated composite materials. Additionally, a locking-free 9-node brick element was integrated into the Chrono framework that makes use of the curvature coordinates at the center of the element. This element is formulated with the Hencky strain measure such that multiplicative finite strain plasticity theory can be used to incorporate soil plasticity models, such as the capped Drucker-Prager failed criterion.&lt;/p&gt;&lt;p&gt;With the shear deformable laminated composite shell element and plastic soil brick element integrated into the Chrono multi-physics simulation engine, an off-road deformable tire and terrain interaction model was developed using the vehicle dynamics simulation module Chrono::Vehicle. An off-road deformable tire model was parameterized based on commercial tire properties and generated as an interchangeable tire model option in the full vehicle dynamics system. Benchmark verification tests were performed to ensure the accuracy of tire deformation and tire force characteristics. Further tests were performed to validate a deformable tire model with a deformable tread pattern constructed from shear deformable shell elements and co-rotational tetrahedral elements. The deformable soil model was also integrated as a terrain option in Chrono::Vehicle and numerical tests were carried out to demonstrate its interaction with rigid and deformable tire models. To make use of the computational performance enhancements available in Chrono, Open Multi-Processing (OpenMP) and Advanced Vector Extensions (AVX) were applied to the evaluation of the elastic force/Jacobian matrix and large matrix operations of flexible bodies, respectively, in order to reduce the computation time by nearly 60%.&lt;/p&gt;

Agrivoltaics offer a sustainable solution to the growing competition between food and energy production. However, their adoption is often constrained by the design and operation challenges associated with optimising the complex trade-off between crop yield and photovoltaic (PV) output. Digital twins can mitigate these risks, yet most agricultural digital twins operate as fragmented digital shadows, lacking high-fidelity modelling, advanced simulation, and bidirectional control capabilities. This study presents a comprehensive, end-to-end digital twin framework to address these limitations. The framework integrates a high-resolution 3D orchard model, reconstructed via UAV photogrammetry, with a CesiumJS-based web interface linked to a modular IoT architecture built on Node-RED, Message Queuing Telemetry Transport (MQTT) protocol and InfluxDB for real-time monitoring and control. A PV simulation engine supports the design, simulation and optimisation of agrivoltaic systems. Bidirectional communication was validated through remote actuation of a physical solar tracker, demonstrating integration among the 3D environment, sensor data and control systems to achieve a closed-loop digital twin. Simulation analyses suggested that panel orientation and row spacing exert a dominant influence on crop-level light distribution. Simulation results demonstrated that a 90° azimuth configuration achieved the highest daily energy yield of 53.97 kWh but reduced peak crop-level irradiance to 205 W/m2. In contrast, the baseline 0° configuration offered a balanced output of 40.86 kWh with a peak light availability of 338 W/m2. The validated, interoperable digital twin architecture provides a reference model for the design, simulation, monitoring and control of an agrivoltaic system, reducing investment uncertainty and supporting sustainable food–energy co-production.

The future of High Performance Computing (HPC) is carved out for all high-performance facilities. As computer processing increases exponentially, especially with the demand for AI training, more speed will equate to more computing density and ultimately more power draw and resource demand. Sandia National Laboratories has not seen or known of anything that would lead us to think this might change in the next 5 to 10 years.

From its founding in 1952 until today, Lawrence Livermore National Laboratory (LLNL) has made significant strategic investments to develop high performance computing (HPC) and its application to national security and basic science. Now, 60 years later, the Computation Directorate and its myriad resources and capabilities have become a key enabler for LLNL programs and an integral part of the effort to support our nation's nuclear deterrent and, more broadly, national security. In addition, the technological innovation HPC makes possible is seen as vital to the nation's economic vitality. LLNL, along with other national laboratories, is working to make supercomputing capabilities and expertise available to industry to boost the nation's global competitiveness. LLNL is on the brink of an exciting milestone with the 2012 deployment of Sequoia, the National Nuclear Security Administration's (NNSA's) 20-petaFLOP/s resource that will apply uncertainty quantification to weapons science. Sequoia will bring LLNL's total computing power to more than 23 petaFLOP/s-all brought to bear on basic science and national security needs. The computing systems at LLNL provide game-changing capabilities. Sequoia and other next-generation platforms will enable predictive simulation in the coming decade and leverage industry trends, such as massively parallel and multicore processors, to run petascale applications. Efficient petascale computing necessitates refining accuracy in materials property data, improving models for known physical processes, identifying and then modeling for missing physics, quantifying uncertainty, and enhancing the performance of complex models and algorithms in macroscale simulation codes. Nearly 15 years ago, NNSA's Accelerated Strategic Computing Initiative (ASCI), now called the Advanced Simulation and Computing (ASC) Program, was the critical element needed to shift from test-based confidence to science-based confidence. Specifically, ASCI/ASC accelerated the development of simulation capabilities necessary to ensure confidence in the nuclear stockpile-far exceeding what might have been achieved in the absence of a focused initiative. While stockpile stewardship research pushed LLNL scientists to develop new computer codes, better simulation methods, and improved visualization technologies, this work also stimulated the exploration of HPC applications beyond the standard sponsor base. As LLNL advances to a petascale platform and pursues exascale computing (1,000 times faster than Sequoia), ASC will be paramount to achieving predictive simulation and uncertainty quantification. Predictive simulation and quantifying the uncertainty of numerical predictions where little-to-no data exists demands exascale computing and represents an expanding area of scientific research important not only to nuclear weapons, but to nuclear attribution, nuclear reactor design, and understanding global climate issues, among other fields. Aside from these lofty goals and challenges, computing at LLNL is anything but 'business as usual.' International competition in supercomputing is nothing new, but the HPC community is now operating in an expanded, more aggressive climate of global competitiveness. More countries understand how science and technology research and development are inextricably linked to economic prosperity, and they are aggressively pursuing ways to integrate HPC technologies into their native industrial and consumer products. In the interest of the nation's economic security and the science and technology that underpins it, LLNL is expanding its portfolio and forging new collaborations. We must ensure that HPC remains an asymmetric engine of innovation for the Laboratory and for the U.S. and, in doing so, protect our research and development dynamism and the prosperity it makes possible. One untapped area of opportunity LLNL is pursuing is to help U.S. industry understand how supercomputing can benefit their business. Industrial investment in HPC applications has historically been limited by the prohibitive cost of entry, the inaccessibility of software to run the powerful systems, and the years it takes to grow the expertise to develop codes and run them in an optimal way. LLNL is helping industry better compete in the global market place by providing access to some of the world's most powerful computing systems, the tools to run them, and the experts who are adept at using them. Our scientists are collaborating side by side with industrial partners to develop solutions to some of industry's toughest problems. The goal of the Livermore Valley Open Campus High Performance Computing Innovation Center is to allow American industry the opportunity to harness the power of supercomputing by leveraging the scientific and computational expertise at LLNL in order to gain a competitive advantage in the global economy.