Hello,
A yearly allocation for the LCRC cluster has been requested with the
following updated information:
Submitter/PI: Daniel Duke
Project Name: XRayFuelSpray
Division: ES
Project title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics
Associated funding: DOE-EERE
Other Systems: None
Science: At the Advanced Photon Source Sector 7-BM at Argonne National Laboratory, we are presently engaged in an ongoing investigation of sprays in high-pressure fuel injection systems using X-ray diagnostic techniques. This work is funded by DOE-EERE. An improved understanding of spray formation physics is essential to the development of more efficient engine technology.
As a part of this work, we rely on high performance computing resources to perform high-fidelity numerical simulations of the experiments, and post-process large experimental data sets.
Over the last few years, the majority of our LCRC allocation has been dedicated to numerical simulations of cavitating flows using OpenFOAM. This work will continue in the following year, however we have been expanding our use of HPC resources at LCRC to encompass new tasks. The scientific objectives of our LCRC allocation have now expanded to include the following:
1) Reconstruction of large, high resolution x-ray tomography datasets,
2) Simulation of cavitating flow experiments using OpenFOAM,
3) Simulation of gasoline direct fuel injection experiments using OpenFOAM,
4) Simulation of high-pressure gaseous jet experiments using Converge,
5) Simulation of diesel direct fuel injection experiments using Converge, and
6) Visualization and analysis of large, high-fidelity data sets.
The tomographic reconstruction sub-task will enable us to perform physically accurate 3D reconstructions from x-ray images of static objects such as fuel injector nozzles and other components with extremely fine resolution (less than 2 microns), over a wide field of view (3 x 2mm). This will be done using an in-house reconstructed code written at APS. The objective of this sub-task is to provide the most precise measurements of nozzle injector geometries possible, which will be used to develop accurate meshes for simulation of those flows. This capability will provide the highest-resolution wall mesh geometry data ever used for such simulations, which will allow us to capture the effects of defects in real samples on the fluid flow. In the last 6 months, we have successfully reconstructed data sets on the order of hundreds of gigabytes on Blues with good parallel efficiency.
The cavitating flows sub-task will continue from prior years’ work. We continue to refine and develop our x-ray radiography and fluorescence experiments at APS, and need to simulate the flow through the test geometries over a wider range of boundary conditions, and with increasing spatial resolution. Over the past few years, a substantial investment has been made in improving the physical sub-models and improving the parallel efficiency of our custom OpenFOAM implementation. Over the next year, we will perform high resolution RANS and LES calculations over a larger parametric space, which will provide a vital comparison with new higher-resolution experimental data. By studying the micron-scale discrepancies between experiments and simulations due to the very small turbulent scales of the flow, we can gain a more complete understanding of which physics are not correctly captured in state-of-the-art sub-models, and understand the physics behind new phenomena observed in the
experiments.
The third sub-task focuses on the simulation of high-pressure gaseous jets from inward opening injectors. In particular, the gaseous flow inside the nozzle sac will be simulated to provide accurate boundary conditions at the nozzle hole exit. Those boundary conditions will be used to describe in detail the characteristics of the under-expanded region and the effect of injection pressure on the shock structure and ultimately on the jet evolution and mixing with surrounding gas. Simulations will be performed with the CFD code Converge and the numerical results will be validated against x-ray radiography data.
The fourth task involves the simulation of gasoline direct injector internal and near-nozzle flows, to compare against experiments conducted at APS. Experimental improvements and upgrades in the next year will allow us to conduct experiments over a wider range of conditions, which will allow us to reach ‘flash boiling’ conditions in the nozzle. Under these conditions, the flow changes substantially due to the violent vaporization of fuel as it exits the nozzle. Understanding the physics behind these changes requires detailed simulation. Our collaborators at the University of Massachusetts-Amherst have been simulating flash-boiling gasoline sprays, and we are currently collaborating with them to implement their new polyhedral meshes and cavitation and flash boiling code for OpenFOAM on Blues and Fusion.
The fifth sub-task involves the simulation of diesel direct injector internal and near-nozzle flows, which will also be compared against experiments being conducted at APS. This work will be conducted in collaboration with researchers from Georgia Tech University. GA Tech is developing new computational approaches to improve the predictive capabilities of spray submodels for use in engine Computational Fluid Dynamics (CFD) codes. X-ray radiography and small angle x-ray scattering measurements at APS will be used to develop and validate a new spray atomization modeling approach for Lagrangian-Eulerian frameworks. The computational approach seeks to appropriately capture the role of liquid turbulence on diesel jet breakup, challenging the widespread adoption of spray submodels that only account for the role of aerodynamic inertial forces on atomization. Simulations will be validated against quantitative measurements from the APS of local fuel mass density and droplet Sauter Mea
n Diameter at matching conditions. A new hybrid turbulence-aerodynamic spray atomization submodel, with improved predictive capabilities for diesel engine simulations, is expected to result from this work.
Finally, we will also use Blues and Fusion to visualize and post-process large data sets. In the last year, we have successfully set up a headless Paraview server on our Fusion dedicated nodes, which can be used to render extremely large OpenFOAM cases and tomography data sets, and send the graphics back to our local workstations for visualization. This level of analysis is not possible on a desktop computer, and substantially increases the efficiency of our data analysis. We intend for this to continue to be a small part of our LCRC allocation. We have recently also implemented the Paraview server on the Blues GPU nodes, which take advantage of the large RAM to handle very large meshes. We intend to recompile the server to take advantage of the GPU cards in order to speed up remote rendering operations.
The addition of several new sub-tasks will require a commensurate increase in our requested allocation. In addition to the dedicated nodes on Fusion that we already operate, we are looking into purchasing dedicated nodes on Blues to accommodate some our increase in HPC resource requirements.
Project description: The cavitation and gasoline spray sub-tasks are being performed using the OpenFOAM framework. These tasks will consume the bulk of the requested core hours. A number of state-of-the-art numerical solvers have been implemented in OpenFOAM; namely incompressible-liquid and fully compressible homogeneous relaxation cavitation models and more recently a fully compressible homogeneous relaxation cavitation model which includes non-condensable gas modeling (HRMFoam). This code can handle both cavitating as well as flash boiling flows. We have successfully implemented both Large Eddy Simulation (LES) and Realizable k-epsilon (RKE) turbulence models. These codes were developed by Schmidt et al at the University of Massachusetts-Amherst, and we continue to collaborate with them on code development and efficiency improvements. The diesel spray measurements will be performed using standard multiphase flow solvers built into OpenFOAM.
We have recently implemented a custom build of OpenFOAM-Extend v3.1 against the system native MPI using mvapich2 on Blues, and very recently against native OpenMPI on Fusion. We are also employing scotch decomposition in order to manage parallel decomposition and load balancing. Scalability studies show excellent parallel efficiency using our custom HRMFoam solver up to 300 cores on Blues and 128 cores on Fusion. The OpenFOAM sub-tasks will consume approximately 150k core hours/quarter in total (less in Q1-2 and more in Q3-4).
The gas jet simulations are performed with the CONVERGE CFD software, its use is already well established at LCRC. This portion of the work will consume 75k core hours in Q1 and Q2.
The diesel spray sub-task in Converge will study turbulent, aerodynamic, and hybrid spray atomization submodel formulations for high-pressure fuel sprays via User Defined Functions tied to the commercial CFD code CONVERGE. The target sprays for simulation will be the Engine Combustion Network (ECN) Spray A and Spray D conditions. LCRC resources will enable us to rapidly assess model formulations and sensitivities over a wide range of liquid Weber and Reynolds number conditions, probing the relevance of turbulence and aerodynamic atomization mechanisms at limiting conditions. This work will require 20-45k core hours per quarter.
Tomographic reconstruction of gasoline direct injection spray measurements from APS will be conducted using the TomoPy software package developed at APS. We run single-node jobs to efficiently perform multiple reconstructions of large (hundred-GB) datasets; these typically require 100 core hours each, but have high RAM requirements.
We have also successfully compiled a Paraview server on our Fusion dedicated nodes which allows us to rapidly visualize the results of our simulations and reconstructions. Using Blues and Fusion will allow us to achieve rapid turnaround of results during beamtime. This is a significant aid to the experimental program, and requires relatively few core hours as it runs in parallel on one node.
Industry partnership: n/a.
Project URL:
Current FY Hours Used: undetermined amount
New FY Requested allocation: 950000
Q1: 250000
Q2: 250000
Q3: 225000
Q4: 225000
Justification: Our requested allocation for next year will exceed 0.5M core-hours in order to encompass new simulation sub-tasks. We wish to note again that we are investigating the option to purchase dedicated nodes on Blues. If we are able to do so, our annual allocation requirements will be reduced by at least 50k core hours per quarter. The allocation requested in this proposal is based on not having dedicated nodes on Blues available.
The majority of core hours will still be dedicated to OpenFOAM simulations, which will be split across both Fusion and Blues, using OpenFOAM-3.1. The tomographic reconstruction and gas jet simulation sub-tasks will be run on Blues only, however we can use more time on Fusion for the OpenFOAM work in order to balance our large core-hour requirement against the higher demand for time on Blues by other projects.
A scalability study of OpenFOAM-3 performance with multi-million cell meshes has been undertaken, demonstrating excellent linear scalability. We observe 81% efficiency relative to single-CPU performance, scaling linearly up to 320 cores / 20 nodes with a 15-million cell mesh (a typical upper limit). We have implemented Scotch decomposition in order to ensure good load balancing & minimization of cross-processor communication boundaries. The present OpenFOAM simulations use static meshes, so load rebalancing at runtime is not a concern. We intend to run typical job sizes of 96 to 128 cores; smaller than the maximum job size for which good scalability has been demonstrated. Further details regarding scalability studies can be obtained by contacting the project PIs.
Storage requirements: In order to reconstruct very large (100 GB+) tomographic data sets, we will require an increase in our project storage allocation to approximately 2.5 TB, if possible. Data will be moved to local storage once processing is completed.
Thank You,
The LCRC Accounts System
