Towards compositional and generative tensor optimizations

With the rapid evolution of large language models (LLM), reinforcement learning (RL) has emerged as a pivotal technique for code generation and optimization in various domains. This paper presents a systematic survey of the application of RL in code optimization and generation, highlighting its role in enhancing compiler optimization, resource allocation, and the development of frameworks and tools. Subsequent sections first delve into the intricate processes of compiler optimization, where RL algorithms are leveraged to improve efficiency and resource utilization. The discussion then progresses to the function of RL in resource allocation, emphasizing register allocation and system optimization. We also explore the burgeoning role of frameworks and tools in code generation, examining how RL can be integrated to bolster their capabilities. This survey aims to serve as a comprehensive resource for researchers and practitioners interested in harnessing the power of RL to advance code generation and optimization techniques.

Stencil computations are a class of algorithms operating on multi-dimensional arrays, which update array elements using their nearest-neighbors. This type of computation forms the basis for computer simulations across almost every field of science, such as computational fluid dynamics, weather simulation, and earthquake prediction. Its mostly regular data access patterns potentially enable it to take advantage of GPU's high computation and data bandwidth. However, manual GPU programming is time-consuming and error-prone, as well as requiring an in-depth knowledge of GPU architecture and programming. This is especially true when the target is high performance. To overcome the difficulties in manual programming, a number of stencil frameworks have been developed to automatically generate GPU codes from user-written stencil code, usually in a Domain Specific Language. The previous stencil frameworks prove that it is feasible to automatically generate GPU codes, but they also introduce a set of unprecedented challenges on generating highly-optimized GPU codes for real stencil applications that may consist of large calculations. This dissertation is based on the Chemora stencil code auto-generation framework, aiming to deal with large calculations existing in real stencil applications. Large stencil calculations usually consist of dozens of grid functions with a variety of stencil patterns, resulting in extremely large code-generation ways to generate code. This dissertation introduces two algorithms to be used in generating highly tuned code. First, we propose to map a calculation into one or more kernels. An algorithm is proposed to optimize the kernel mapping by improving thread-level parallelism and minimizing off-chip memory accesses. Next, we propose an efficiency-based buffering algorithm which operates by scoring a change in buffering strategy for a grid function (GF) using a performance estimation and resource usage. With the algorithm, a near optimal solution can be found in (b-1)N(N+1)/2 steps, instead of b^N steps, for a calculation with N GFs and b buffering strategies. The current Chemora framework supports six (i.e. b = 5) buffering strategies. Finally, an analytic performance model is proposed to predict the execution time of a code generation way. In addition, we wrote a set of micro-benchmarks to explore and measure some performance-critical GPU microarchitecture features and parameters for better performance modeling.

On the Use of a Many-core Processor for Computational Fluid Dynamics Simulations

One the of the significant challenges faced in simulating atmospheric boundary layers (ABLs) with standard computational fluid dynamics (CFD) models is to preserve the correct ABL profiles throughout the flow domain. Appropriate ABL profiles may be imposed at the inlet to the CFD domain but they are often progressively modified downwind by the CFD model until they are no longer representative of the correct stability class and/or wind speed. To address this issue, this paper reviews solutions proposed in the literature with respect to the choice of turbulence model and CFD boundary conditions (inlet profiles, wall conditions, outlet and top boundary conditions). The main focus is on modelling stable ABLs which often produce the largest dispersion distances in hazard analyses. CFD results are presented for a 2 km long domain with flat terrain using ANSYS-CFX with standard and modified k-e turbulence models and two different choices of inlet profiles. The results show that improvements in the ABL profiles are obtained using a modified turbulence model with a consistent choice of inlet profile.

This paper describes an intermediate language to be generated by a syntax analyzer and processed by a code generator. An (essentially) optional code optimization phase may be used before code generation. The language is designed to exclude source and target dependencies (these are specified in a set of auxiliary tables) and has been used to implement a general purpose code optimization module. This module has been incorporated into compilers for several source languages and has resulted in production-quality target code.The paper focuses on the features of the language and reviews the optimizations for which it has been designed. A major benefit of this approach has been to isolate many of the characteristics which lead to code optimization opportunities. Of particular note are numerous facts which can be gathered in a target-independent manner, even though their utilization will depend on the details of the target machine.

A new model of computational fluid dynamics (CFD)-based algorithm for coronary CT angiography (CCTA)-derived fractional flow reserve (FFR) (CT-FFR) analysis by expanding the coronary tree to smaller-diameter lumen (0.8 mm) using Newton-Krylov-Schwarz (NKS) method to solve the three-dimensional time-dependent incompressible Navier-Stokes equations has been developed; however, the diagnostic performance of this new method has not been sufficiently investigated. The aim of this study was to determine the diagnostic performance of a novel CT-FFR technique by expanding the coronary tree in the CFD domain. Six centres enrolled 338 symptomatic patients with suspected or known coronary artery disease (CAD) who prospectively underwent CCTA and FFR. Stenosis assessment in CCTA and CT-FFR analysis were performed in independent core laboratories. Haemodynamically significant stenosis was defined by a CT-FFR and FFR ≤ 0.80, and anatomically obstructive CAD was defined as a CCTA with stenosis ≥ 50%. Diagnostic performance of CT-FFR was evaluated against invasive FFR using receiver operating characteristic (ROC) curve analysis. The correlation between CT-FFR and invasive FFR was analysed using the Spearman correlation coefficient and Bland-Altman analysis. Intra-observer and inter-observer agreements were evaluated utilizing the intraclass correlation coefficient (ICC). In this study, 338 patients with 422 targeted vessels were investigated, revealing haemodynamically significant stenosis in 31.1% (105/338) of patients and anatomically obstructive stenosis in 54.1% of patients. On a per-vessel basis, the area under the ROC curve for CT-FFR was 0.94 vs. 0.76 for CCTA (P < 0.001). Per-vessel accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were 89.8%, 89.3%, 90.0%, 79.0%, and 99.2%, respectively, for CT-FFR and were 68.4%, 82.8%, 62.3%, 48.1%, and 89.6%, respectively, for CCTA stenosis. CT-FFR and FFR were well correlated (r = 0.775, P < 0.001) with a Bland-Altman bias of 0.0011, and limits of agreement from -0.1509 to 0.1531 (P = 0.770). The ICCs with CT-FFR for intro- and inter-observer agreements were 0.919 (95% CI: 0.866-0.952) and 0.909 (95% CI: 0.851-0.945), respectively. The average computation time for CT-FFR analysis was maintained at 11.7 min. This novel CT-FFR model with the inclusion of smaller lumen provides high diagnostic accuracy in detecting haemodynamically significant CAD. Furthermore, the integration of the NKS method ensures that the computation time remains within an acceptable range for potential clinical applications in the future.

Succinct, declarative, and domain-specific modeling languages have many advantages when creating simulation models. However, it is often challenging to efficiently execute models defined in such languages. We use code generation for model-specific simulators. Code generation has been successfully applied for high-performance algorithms in many application domains. By generating tailored simulators for specific simulation models defined in a domain specific language, we get the best of both worlds: a succinct, declarative and formal presentation of the model and an efficient execution. We illustrate this based on a simple domain-specific language for biochemical reaction networks as well as on the network representation of the established BioNetGen language. We implement two approaches adopting the same simulation algorithms: one generic simulator that parses models at runtime and one generator that produces a simulator specialized to a given model based on partial evaluation and code generation. Akin to profile-guided optimization we also use dynamic execution of the model to further optimize the simulators. The performance of the approaches is carefully benchmarked using representative models of small to mid-sized biochemical reaction networks. The generic simulator achieves a performance similar to state of the art simulators in the domain, whereas the specialized simulator outperforms established simulation algorithms with a speedup of more than an order of magnitude. This repository contains the code generation software as described in the 2020 PADS paper Partial evaluation via code generation for static stochastic reaction network models.

This paper describes a system that accepts compact specifications of an intermediate code and target machine and produces program code for an integrated code generator and peephole optimizer. A compiler for most of C uses this packa.ge. It emits code comparable to PCCI’S, but it runs over five times faster on preliminary benchmarks. This compiler also runs over twice as fast as a version of pcc2 with a hand-coded, VAX-specific code generator. The code generators are produced as follows. A programmer describes a naive code generator by means of a non-procedural specification. The programmer also prepares a machine description for a retargetable peephole optimizer [2]. These two systems are used together to compile a testbed, and the compiler records each peephole optimization as it is made. This record and the specification of the naive code generator are compiled into a fast, integrated code generator and optimizer. This production code generator then takes the place of the slower “training” version. The production code generator and optimizer are integrated to the point that the code to be generated is communicated from one to the other by encoding it in the program counter, which obviates most inter-phase communication costs. Interpretive peephole optimizers have been driven by traces from retargetable peephole optimizers [3] and integrated with interpretive code generators [4], but the current work is distinguished by the production of a hard-coded, optimizing code generator. Historically, retargetable code generators (i.e., those not largely rewritten for each new machine)

Einstein summation for multidimensional arrays

The rapid evolution of Artificial Intelligence (AI) has brought about significant advancements in multiple domains, including software development. One of the most promising innovations is AI-powered code generation through Large Language Models (LLMs), such as OpenAI’s GPT-3 and GPT-4. These models, having been trained on large amounts of programming data, have the ability to produce human-readable code from natural language inputs, which is a big potential for simplifying and optimizing software development processes. The aim of this paper is to analyze the performance of LLMs in automated software development by testing their performance on a variety of tasks such as code generation, debugging, and optimization of software. The research explores both the strengths and weaknesses that these models have to offer, in terms of some of the most important indicators like code quality, generation time, and maintainability of the code. According to our observation, although LLMs hold immense potential to automate mundane programming tasks and enhance developer productivity, they still struggle to cope with more intricate, domain-specific programming tasks involving a higher level of understanding, for example, designing architectures and top-level decision-making. In spite of such shortcomings, LLMs can tremendously enhance software development processes, particularly for small-scale projects or act as helpers for more senior developers. The paper summarizes by reflecting on the potential for LLMs to transform software development processes in the future, while also the importance of the model's reliability, coding quality, and security to be improved if it is to be made applicable to larger, more crucial uses.

Nowadays, a variety of Big Data architectures are emerging to organize the Big Data life cycle. While some of these architectures are proposed for general usage, many of them are proposed in a specific application domain such as smart cities, transportation, healthcare, and agriculture. There is, however, a lack of understanding of how and why Big Data architectures vary in different domains and how the Big Data architecture strategy in one domain may possibly advance other domains. Therefore, this paper surveys and compares the Big Data architectures in different application domains. It also chooses a representative architecture of each researched application domain to indicate which Big Data architecture from a given domain the researchers and practitioners may possibly start from. Next, a pairwise cross-domain comparison among the Big Data architectures is presented to outline the similarities and differences between the domain-specific architectures. Finally, the paper provides a set of practical guidelines for Big Data researchers and practitioners to build and improve Big Data architectures based on the knowledge gathered in this study.

Code optimization is an intricate task that is getting more complex as computing systems evolve. Managing the program optimization process, including the implementation and evaluation of code variants, is tedious, inefficient, and errors are likely to be introduced in the process. Moreover, because each platform typically requires a different sequence of transformations to fully harness its computing power, the optimization process complexity grows as new platforms are adopted. To address these issues, systems and frameworks have been proposed to automate the code optimization process. They, however, have not been widely adopted and are primarily used by experts with deep knowledge about underlying architecture and compiler intricacies. This article describes the requirements that we believe necessary for making automatic performance tuning more broadly used, especially in complex, long-lived high-performance computing applications. Besides discussing limitations of current systems and strategies to overcome these, we describe the design of a system that is able to semi-automatically generate efficient platform-specific code. In the proposed system, the code optimization is programmer-guided, separately from application code, on an external file in what we call optimization programming. The language to program the optimization process is able to represent complex collections of transformations and, as a result, generate efficient platform-specific code. A database manages different optimized versions of code regions, providing a pragmatic approach to performance portability, and the framework itself has separate components, allowing the optimized code to be used on systems without installing all of the modules required for the code generation. We present experiments on two different platforms to illustrate the generation of efficient platform-specific code that performs comparable to hand-optimized, vendor-provided code.

Aerodynamic shape optimisation technology is presented, comprising an efficient variable fidelity shape parameterisation method, an efficient andhigh quality mesh deformation scheme, and a parallel optimisation algorithm. The objective of the research presented here is the comparison of truly three-dimensional optimisations of aircraft wings in both aerodynamic and aeroelastic environments. The novel shape parameterisation technique allows various fidelities of design parameters, ranging from detailed surface changes to novel truly three-dimensional planform adjustments. An efficient interpolation scheme, using radial basis functions, transfers domain element movements into direct deformations of the design surface and corresponding CFD mesh, thus allowing total independence from the grid generation package and type (structured or unstructured). Optimisation is independent from the CFD flow solver by obtaining sensitivity information for an advanced parallel gradient-based optimiser by finite-differences. This ‘wrap-around’ optimisation technique is applied to a modern large transport aircraft wing in the cruise flight condition for minimum drag with stringent constraints in lift, volume, and two root moments. The objective of all optimisations is aerodynamic, however the static aeroelastic deflection provided by an aeroelastic solver will give that particular optimisation increased accuracy and real world relevance. The result of a constrained inviscid aerodynamic optimisation is presented and has a significant reduction in drag when compared to the initial wing with no violation of any constraints. The shape parameterisation method demonstrates that only a low number of design variables are necessary to achieve innovative planform and surface geometries with dramatically improved performance. Computational fluid dynamics (CFD) methods are now commonplace in aerospace industries, and at the forefront of analysis capabilities, providing a fast and effective method of predicting a design’s aerodynamic performance. However, with ever increasing complexity of designs, engineers can often struggle to interpret the intricacies of the CFD results sufficiently to be able to manually alter the geometry to improve performance. Hence, there has been an increase in demand for intelligent and automatic shape optimisation schemes. This requires combining geometry control methods with numerical optimisation algorithms, to provide a mechanism to mathematically seek improved and optimum designs, using CFD as the analysis tool. Optimisation requires consideration of three issues, each of which have numerous solutions: shape parameterisation including CFD surface and volume mesh deformation, computation of design variable derivatives, and effective use of these derivatives to improve design. Geometry parameterisation is critical for effective shape optimisation. This is the method of representing the design surface, and defines the degrees of freedom in which the geometry can be altered and, ideally, this should be linked with an effective method of deforming the CFD surface and volume mesh in a corresponding fashion. Parameterising complex shapes is a problem that remains a serious obstacle to both manual and automatic CFD-based optimisation. A wide variety of shape control and morphing methods have been developed, but many do not allow sufficiently

Because of the complexity of the flow within light water reactor (LWR) cores, numerous small-scale phenomena locally influence heat transfer and critical heat flux (CHF). They include development of viscous and thermal boundary layers, interchannel mixing, spacer grid mixing, rod vibrations, or confinement effects such as the proximity of the walls or the influence of the gap between adjacent fuel bundles. LWR prototypical conditions are particularly harsh environments and limit measurements to quantities such as pointwise pressure drop and temperature, the latter resulting in global heat transfer and CHF correlations. The local phenomena mentioned above are embedded in these correlations, leading to inherent empiricism (and therefore conservatism). Validated computational fluid dynamics (CFD) codes and models can predict these phenomena, thus providing modelization tools of greater accuracy. However, major requirements for validation campaigns include the matching of validation and application domains and the deployment of mature and high-resolution diagnostics. For the latter, many are available for single-phase flows due to their predominance in several research fields. Furthermore, in the lower part of LWR cores, flow is single phase, and only this regime is considered in this paper. To circumvent the challenges of deploying diagnostics in LWR conditions, surrogate fluids are commonly used, enabling the measurement of velocity, temperature, pressure, or wall shear stress. A large number of single-phase tests with resolution adequate to validate CFD models have been conducted with air, steam, and water at moderate temperature and pressure. However, to date, with these fluids, the application domain defined by the Reynolds and Prandtl numbers has not been reached.Four surrogate gases are proposed to match application and validation domains while allowing the deployment of a broad range of diagnostics: pressurized sulfur hexafluoride, xenon, cryogenic nitrogen, and highly pressurized air. By controlling their operating temperature and pressure, they allow matching prototypical Reynolds and Prandtl numbers while preserving the length scale, velocity scale, and timescale. This is achieved by reproducing the kinematic viscosity and thermal diffusivity of several nuclear reactor coolants. Furthermore, for single-phase conjugate heat transfer, a complete scaling analysis is performed for one pressurized water reactor fuel rod within a bundle under normal operating conditions. Electrically heated rods made of magnesium oxide and Zircaloy, combined with the proposed surrogate fluids, provide a close matching of conjugate heat transfer. Additionally, the use of these surrogates offers a significant decrease of the heating and pumping powers. Single-phase heat transfer separate-effect tests can then be performed for the first time in a laboratory setup with one, or several, full-size fuel bundles at prototypical conditions, while allowing the deployment of a large range of diagnostics. Finally, existing test facilities for hydraulics and thermal hydraulics of prototypical fuel bundles can be utilized with minor retrofits, further facilitating test implementation.

The structure and memory organization of graphics processor units (GPUs) manufactured by NVIDIA and the use of CUDA programming technology to solve computational fluid dynamics (CFD) problems is reviewed and discussed. The potential of using a general-purpose GPU to solve fluid dynamics problems is examined. The code optimization with the utilization of various memory types is considered. Some CFD benchmark problems focused on simulation of viscous incompressible fluid flows are solved on GPUs. Consideration is given to the application of the finite volume method and projection method. Programming implementation of various components of the computational procedure, solution of Poisson equation for pressure and multigrid method to solve the system of algebraic equations, is provided. By using meshes of varying resolutions and different techniques for dividing up the input data into blocks, the speedup of the GPU solution is compared to the CPU approach.