Hello,
A yearly allocation for the LCRC cluster has been requested with the
following updated information:
Submitter/PI: Chris Powell
Project Name: XRayFuelSpray
Division: ES
Project title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics
Associated funding: DOE-EERE Vehicle Technologies Office
Other Systems: None
Science: At the Advanced Photon Source Sector 7-BM at Argonne National Laboratory, we are engaged in an ongoing investigation of sprays in high-pressure fuel injection systems using X-ray diagnostic techniques. An improved understanding of spray breakup and fuel/air mixing is essential to the development of cleaner, more efficient combustion engines, and is an important goal of the DOE-EERE Vehicle Technologies Office.
As a part of this work, we rely on high performance computing resources to perform high-fidelity numerical simulations of the experiments, and post-process large experimental data sets. Simulations are performed in collaboration with several groups outside of Argonne, including research groups at UMass-Amherst, Georgia Tech, and Monash University.
The scientific objectives of our LCRC request include several Tasks.
- Simulation of nozzle flow experiments using OpenFOAM
- Development of new models for diesel spray breakup and combustion
- Simulations of flow in real-world fuel injectors
- Reconstruction of large, high resolution x-ray tomography datasets
- Visualization and analysis of large, high-fidelity datasets
Project description: The Project Description is broken down by Task
Simulation of nozzle flow experiments using OpenFOAM
We continue to refine and develop our x-ray radiography and fluorescence measurements of cavitating flows, and need to simulate the flow through the test geometries over a wider range of boundary conditions, with increasing spatial resolution. Over the past few years, a substantial investment has been made in improving the physical sub-models and improving the parallel efficiency of our custom OpenFOAM implementation (HRMFoam) in collaboration with UMass. High resolution is necessary in order to capture small scale fluctuations at the same resolution as the x-ray experiments. Typically, simulations of this kind are not resolved below 10 microns; this is unsuitable for comparison to our experimental data. In the coming FY we will continue to run simulations comparing both realizable k-epsilon Reynolds-averaged (RANS) and large eddy simulation (LES) turbulence models over a range of geometry sizes and pressure conditions. The high fidelity time-resolved LES data are unique, as
they provide an indication of the temporal fluctuations in the flow which can be directly compared to our time resolved x-ray radiography data (which has a resolution of 5 micron at 150 ns). The experiments suggest that turbulence is strongly coupled to cavitation, however this effect is not yet well understood because most simulations of these flows that have been covered in the literature are temporally averaged (i.e. RANS), and those LES that have been done had insufficient resolution. By studying the micron-scale discrepancies between experiments and simulations due to the very small turbulent scales of the flow at sub microsecond time scales, we can gain a more complete understanding of which physics incorrectly captured in state-of-the-art sub-models, and understand the physics behind new phenomena observed in the experiments.
Development of new models for diesel spray breakup and combustion
Georgia Tech is developing a new open-source engine CFD code architecture based on OpenFOAM, utilizing new sub-modeling approaches that will improve the predictive capability of engine simulations. X-ray radiography and small angle x-ray scattering measurements from Argonne will be used to develop and validate a new spray atomization modeling approach for Lagrangian-Eulerian frameworks. The computational approach seeks to appropriately capture the role of liquid turbulence on diesel jet breakup, challenging the widespread adoption of spray sub-models that only account for the role of aerodynamic inertial forces on atomization. Simulations will be validated using Argonne’s quantitative measurements of local fuel mass density and droplet Sauter Mean Diameter at matching conditions. A new hybrid turbulence-aerodynamic spray atomization sub-model, with improved predictive capabilities for diesel engine simulations, is expected to result from this work.
To further validate and demonstrate the improved spray sub-models, Georgia Tech will conduct reacting simulations to be validated against the Engine Combustion Network’s larger “Spray A” measurement database. Sub-model choices were assessed against validation data sets in the past year, and they will be into the OpenFOAM architecture during FY2018.
Simulations of flow in real-world fuel injectors
We propose to use LCRC resources to study nozzle flows and sprays, to advance the understanding of factors which influence fuel spray characteristics. To perform these studies, we will continue developing techniques for modeling fuel injection at low computational cost, while still capturing essential information about the flow. Establishing best practices will make it possible for CFD to be used not only for scientific study of these flows, but as an automotive design tool.
HRMFoam has been developed to model fuel injection and spray development. Our present focus is the coupled simulation of flows through several diesel and gasoline injectors. The Engine Combustion Network (ECN), a collaborative effort to advance the understanding of fuel injection has selected a number of injector geometries and target operating conditions so that experimentalists and modelers can produce comparable work and establish best practices. Since our goals are drawn from these geometries and conditions, it will be possible to validate our results against existing and future experimental studies of these nozzles. In addition, since other modelers are also using these target conditions, it will also be possible to compare our results against simulations performed using different modeling techniques.
Simulations in this Task will focus on experimental data from several sources:
1. The Spray Combustion Consortium is a joint research project involving both Argonne, Sandia, and UMass. In the course of this research, we simulate internal fuel injector flows for research prototypes developed at Sandia National Laboratory and imaged at the Argonne. We expect to complete a run matrix of 12 cases.
2. The Engine Combustion Network is an international effort to study fuel injectors both experimentally and computationally. While the ANL Advanced Photon Source provides requisite inputs and validation for our computation, we plan to simulate the internal details of the fuel injection process. We will simulate Spray G (10 cases) and the new multi-hole targets (4 cases). The multi-hole targets will require more resources, but there will be fewer cases.
An important part of this Task will be scalability improvements on KNL nodes. The KNL architecture promises to be very efficient and scalable for our kinds of computation. Because of the unstructured nature of data storage, the CPU has little ability to anticipate what data it needs next. Consequentially, the time required for fetches to main memory is likely to be a bottleneck in code performance. We hope to take advantage of the MCDRAM of the KNL nodes in order to speed up our code. We believe that in our previous scalability tests, however, the use of a sub-par MPICH implementation is slowing down the parallel execution of the code. We are currently performing tests with an MPICH specifically compiled for the actual interconnects of the cluster. These performance and scalability studies, when performed on Bebop, will require numerous large, but short, runs.
Reconstruction of large, high resolution x-ray tomography datasets
This Task enables us to perform physically accurate 3D reconstructions from x-ray images of static objects such as fuel injector nozzles and other components with extremely fine resolution (less than 2 microns), over a wide field of view (3 x 2mm). This is done using tomoPy, an Argonne-authored open source reconstruction code. The objective of this Task is to generate high precision measurements of nozzle injector geometries, which are then used to develop accurate meshes for CFD simulation of those flows. This capability will provide the highest-resolution wall mesh geometry data ever used for such simulations, allowing us to capture the effects on the fluid flow of features such as defects and surface finish in real samples.
The major benefit of being able to do this work on LCRC machines rather than desktop computers is fast turnaround time; reconstructions in 15-20 minutes instead of hours. This allows us to assess the quality of the data on-the-fly and make rapid adjustments to our experiments. In order to overcome queue wait time issues associated with these fast-turnaround jobs, we are currently utilizing our dedicated nodes on Blues. This task will benefit substantially from the availability of the more powerful Haswell nodes and we will investigating the possibility of running these tasks on Bebop in the coming year.
Visualization and analysis of large, high-fidelity datasets
We have successfully set up a headless Paraview server on Blues which can be used to render extremely large OpenFOAM cases and tomography data sets, and send the graphics back to our local workstations for visualization. This allows us to take advantage of Blues’ high-RAM nodes to look at very large datasets. This level of analysis is not possible on a desktop computer, and substantially increases the efficiency of our data analysis. We will use our group’s dedicated nodes on Blues to continue this activity, and investigate the possibility of implementing this capability on Bebop. We intend for this to continue to be a small part of our LCRC allocation.
Industry partnership: None
Project URL:
Current FY Hours Used: undetermined amount
New FY Requested allocation: 500000
Q1: 125000
Q2: 125000
Q3: 125000
Q4: 125000
Justification: Since we do not yet have access to Bebop, we have identified similar hardware at the Texas Advanced Supercomputing Center (TACC) and San Diego Supercomputing Center (SDSC) for testing. The performance of HRMFoam on Intel Xeon-E5 processors has been tested on the TACC Stampede cluster architecture, with good scalability up to 100 processors.
Preliminary KNL scaling tests were performed on a pre-production test cluster on TACC's Stampede 2 system, which contained pre-release software. As such, HRMFoam was built using a test version of MVAPICH2, which was not fully verified for KNL. TACC has since put Stampede 2 into production, along with updated software stacks. The MVAPICH2 library was not included in the new system, suggesting it may not have performed well during tests. As such, work on compiling the foam-extend branch of the OpenFOAM library using Intel MPI instead of MVAPICH2 is underway. Once successful, this should yield significant scaling improvements.
Storage requirements: In order to reconstruct very large (i.e. 100 GB) tomographic data sets and store high-resolution CFD fields for post processing, we require typically 5 TB of storage space. Our current storage allocation is adequate. Data will be moved to local storage once processing is completed.
Thank You,
The LCRC Accounts System
