Hello,
A yearly allocation for the LCRC cluster has been requested with the
following updated information:
Submitter/PI: Daniel Duke
Project Name: XRayFuelSpray
Division: ES
Project title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics
Associated funding: DOE-EERE
Other Systems: None
Science: At the Advanced Photon Source Sector 7-BM at Argonne National Laboratory, we are presently engaged in an ongoing investigation of sprays in high-pressure fuel injection systems using X-ray diagnostic techniques. This work is funded by DOE-EERE. An improved understanding of spray formation physics is essential to the development of more efficient engine technology.
As a part of this work, we rely on high performance computing resources to perform high-fidelity numerical simulations of the experiments, and post-process large experimental data sets.
The scientific objectives of our LCRC request include the following sub-projects:
- Simulation of cavitating flow experiments using OpenFOAM,
- Simulation of diesel direct fuel injection experiments using OpenFOAM,
- Reconstruction of large, high resolution x-ray tomography datasets, and
- Visualization and analysis of large, high-fidelity datasets.
The cavitating flows sub-task will continue from prior years’ work. We continue to refine and develop our x-ray radiography and fluorescence experiments at APS, and need to simulate the flow through the test geometries over a wider range of boundary conditions, and with increasing spatial resolution. Over the past few years, a substantial investment has been made in improving the physical sub-models and improving the parallel efficiency of our custom OpenFOAM implementation. In the previous FY, we were able to undertake some of the largest simulations of these flows performed to date (upward of 250,000 core hours on a 135 million cell grid with 3 micron minimum cell size, across a 500 micron diameter geometry). The high resolution is necessary in order to capture small scale fluctuations at the same resolution as the x-ray experiments. Typically, simulations of this kind are not resolved below 10 microns; this is unsuitable for comparison to our experimental data. In the fo
llowing FY we will continue to run simulations comparing both realizable k-epsilon Reynolds-averaged (RANS) and large eddy simulation (LES) turbulence models over a range of geometry sizes and pressure conditions. The high fidelity time-resolved LES data are unique, as they provide an indication of the temporal fluctuations in the flow which can be directly compared to our time resolved x-ray radiography data (which has a resolution of 5 micron at 150 ns). The experiments suggest that turbulence is strongly coupled to cavitation, however this effect is not yet well understood because most simulations of these flows that have been covered in the literature are temporally averaged (i.e. RANS), and those LES that have been done had insufficient resolution. By studying the micron-scale discrepancies between experiments and simulations due to the very small turbulent scales of the flow at sub microsecond time scales, we can gain a more complete understanding of which physics are
not correctly captured in state-of-the-art sub-models, and understand the physics behind new phenomena observed in the experiments.
The second sub-task involves the simulation of diesel direct injector internal and near-nozzle flows, which will also be compared against experiments being conducted at APS. GA Tech is developing new computational approaches to improve the predictive capabilities of spray sub-models for use in engine Computational Fluid Dynamics (CFD) codes. X-ray radiography and small angle x-ray scattering measurements at APS will be used to develop and validate a new spray atomization modeling approach for Lagrangian-Eulerian frameworks. The computational approach seeks to appropriately capture the role of liquid turbulence on diesel jet breakup, challenging the widespread adoption of spray sub-models that only account for the role of aerodynamic inertial forces on atomization. Simulations will be validated against quantitative measurements from the APS of local fuel mass density and droplet Sauter Mean Diameter at matching conditions. A new hybrid turbulence-aerodynamic spray atomization
sub-model, with improved predictive capabilities for diesel engine simulations, is expected to result from this work. A secondary goal in this sub-task focuses on the application of the GA Tech atomization sub-models to vaporizing and reacting conditions. GA Tech is developing a large-eddy simulation (LES) framework to study the influence of spray-induced turbulence structures and the resultant mixing field on diesel combustion. New vaporizing spray measurements from APS will be used for validation, along with combustion measurements from the GA Tech High Pressure and Temperature (HiPT) Spray Chamber. The results of this work will yield insights into the link between near-nozzle spray development and downstream flame evolution. A new Lagrangian-Eulerian LES modeling framework for diesel sprays, with improved spray and turbulence-chemistry interaction sub-models, is expected to result from this work.
The third sub-task in this project is tomographic reconstruction of high resolution x-ray CT datasets. This ongoing work enables us to perform physically accurate 3D reconstructions from x-ray images of static objects such as fuel injector nozzles and other components with extremely fine resolution (less than 2 microns), over a wide field of view (3 x 2mm). This is done using tomoPy, an in-house reconstruction code written at APS. The objective of this sub-task is to provide the most precise measurements of nozzle injector geometries possible, which will be used to develop accurate meshes for simulation of those flows. This capability will provide the highest-resolution wall mesh geometry data ever used for such simulations, which will allow us to capture the effects of defects in real samples on the fluid flow. The major benefit of being able to do this work on Blues rather than on desktop computers is fast turnaround time; reconstructions in 15-20 minutes instead of hours.
This allows us to assess the quality of the data on-the-fly and make rapid adjustments to our experiments. In order to overcome queue wait time issues associated with these fast-turnaround jobs, we are investigating the possibility of purchasing dedicated nodes.
Finally, we will also use Blues to visualize and post-process large data sets. We have successfully set up a headless Paraview server which can be used to render extremely large OpenFOAM cases and tomography data sets, and send the graphics back to our local workstations for visualization. This allows us to take advantage of Blues’ high-RAM nodes to look at very large datasets. This level of analysis is not possible on a desktop computer, and substantially increases the efficiency of our data analysis. We intend for this to continue to be a small part of our LCRC allocation.
Several sub-tasks that were conducted during the previous FY have now concluded, and our sub-task on simulation of cavitating flows is being scaled back in the coming FY. As such, our allocation request has been reduced relative to the previous FY. Since our dedicated Fusion nodes are being taken offline, we are looking into purchasing dedicated nodes on Bebop to accommodate some our increase in HPC resource requirements, and particularly to deal with the issue of queue wait times for APS data processing jobs that require fast turnaround.
Project description: The cavitation and diesel spray modeling sub-tasks are being performed using the OpenFOAM framework. These tasks will consume the bulk of the requested core hours. A number of state-of-the-art numerical solvers have been implemented in OpenFOAM; namely incompressible-liquid and fully compressible homogeneous relaxation cavitation models and more recently a fully compressible homogeneous relaxation cavitation model which includes non-condensable gas modeling (HRMFoam). This code can handle both cavitating as well as flash boiling flows. These codes were developed by Schmidt et al at the University of Massachusetts-Amherst, and we continue to collaborate with them on code development and efficiency improvements. The HRMFoam solver has been tested for scalability and shown good performance on up to 300 processors. We typically run jobs with 64 to 128 processors.
The diesel spray sub-task in OpenFOAM studies turbulent, aerodynamic, and hybrid spray atomization sub-model formulations for high-pressure fuel sprays via customized sub-models. The target sprays for simulation will be the Engine Combustion Network (ECN) non-vaporizing Spray A and Spray D conditions. LCRC resources will enable GA Tech to rapidly assess model formulations and sensitivities over a wide range of liquid Weber and Reynolds number conditions, probing the relevance of turbulence and aerodynamic atomization mechanisms at limiting conditions.
The combusting spray sub-task in OpenFOAM investigates the coupling of spray sub-models and turbulence-chemistry interactions for spray combustion through the development and implementation of a Large-Eddy Simulation (LES) framework. The target sprays for simulation will be the Engine Combustion Network (ECN) reacting Spray A condition. LCRC resources facilitate the simulation of multiple LES realizations in order to obtain reliable statistics for predicted spray and combustion parameters, which is essential for comparison against experimental measurements from APS and GA Tech. These comparisons will enable an in-depth study of both the instantaneous and average coupling between spray-induced turbulence structures and the resultant chemistry and flame evolution.
The tasks described above will be run on 64 to 128 processors and will consume the bulk of the allocation requirement. Due to the conclusion of several sub-tasks from the previous FY, our allocation requirement for FY17 is reduced.
Tomographic reconstruction of APS data will be conducted using the TomoPy software package developed at Argonne. We run single-node jobs to efficiently perform multiple reconstructions of large (hundred-GB) datasets; these typically require less than 100 core hours each, but have high RAM requirements. This task will benefit substantially from the availability of the more powerful Haswell nodes and we are investigating the possibility of purchasing dedicated nodes on the new Bebop machine for this purpose.
We have also successfully compiled a Paraview server on Blues which allows us to rapidly visualize the results of our simulations and reconstructions without having to wait to download the data to our local computer. This allows us to achieve rapid turnaround of results during beamtime. This is a significant aid to the experimental program. The server runs typically on a single Blues ‘biggpu’ node (16 cores) which allows us to open simulation datasets that are much larger than the RAM available on our desktop computers. This task requires relatively few core hours, since the server is only used for a few hours at a time.
Industry partnership: n/a.
Project URL:
Current FY Hours Used: undetermined amount
New FY Requested allocation: 500000
Q1: 125000
Q2: 125000
Q3: 125000
Q4: 125000
Justification: Due to the conclusion of several sub-tasks, and plans to purchase dedicated nodes, our allocation requirements for FY17 are much less than in the previous year, and will not exceed 0.5M core hours.
Storage requirements: In order to reconstruct very large (i.e. 100 GB) tomographic data sets and store high-resolution CFD fields for post processing, we require typically 5 TB of storage space. Our current storage allocation is adequate. Data will be moved to local storage once processing is completed.
Thank You,
The LCRC Accounts System
