Ramesh Balakrishnan,
Thank you for your project request. It has been sent to the LCRC
Allocations Committee for consideration. You will be notified when
a decision is made. Below is a copy of your request, for your records:
Applicant's Name: Ramesh Balakrishnan
Applicant's Division: LCF
Project Name: 3DDES
Project Title: Detached Eddy Simulation of Separated High Reynolds Number Flows
Associated Funding: ASCR Funding via ALCF
Other Systems: Theta, Mira, Bebop
Science: I am requesting 10M core hours, for the whole year, on Bebop, to develop detached eddy simulation models that can serve as a wall model for moderately separated flow in thick boundary layers. Examples of such flow are the atmospheric
boundary layer (ABL) over complex terrain, large eddy breakup devices (LEBU), and separated flow past airfoils at high angles of attack (i.e. greater than 10 degrees).
Project Description: Preliminary simulations of the ABL over Complex Terrain have been carried out on Bebop during the Early User Period, where the PI had an opportunity to run and test out modified version(s) of OpenFOAM, SU2, and FDL3DI on Bebop. The
simulations were run on meshes ranging from 28M grid points to 140M grid points, and on a maximum of 4096 cores of Bebop. The codes make use of MPI and can optionally use OpenMP. In addition, one can vectorize the code for the KNL nodes (in which case OpenMP
is not needed at all). During the early user period, the vectorization and OpenMP options were not tested out. The flat MPI codes do scale well on both the KNL and Broadwell nodes, although the time-to-solution is about five to six times slower on the KNL
nodes (when compared to the Broadwell nodes). This is something that I plan to investigate in more detail as part of this allocation request.
Industry Partnership: None at this time.
Project URL:
Requested allocation: 10000000
for Q1: 2500000
for Q2: 2500000
for Q3: 2500000
for Q4: 2500000
Efficiency: modified OpenFOAM: 35,000 finite volumes per core (on smallest number of cores)
=======================================
cores time/time_step
4096 (KNL) 27 seconds (64 cores/node x 64 nodes)
2048 (KNL) 43 seconds (64 cores/node x 32 nodes)
1024 (KNL) 75 seconds (64 cores/node x 16 nodes)
4096 (BDW) 2 seconds (32 cores/node x 128 nodes)
2048 (BDW) 5 seconds (32 cores/node x 64 nodes)
Please note that the 10M core hours (for the whole year) is requested on Bebop.
1024 (BDW) 12 seconds (32 cores/node x 32 nodes)
modified SU2: 12,000 finite volumes per core (on smallest # of cores)
=================================
solver cores time/time_step
FV 1024 (KNL) 11 seconds
FE_DG (p4) 1024 (KNL) 16 seconds (same # finite volumes/core)
FV 2048 (KNL) 8 seconds
FE_DG (p4) 2048 (KNL) 11 seconds (same # finite volumes per core)
Storage Requirements: 5TB
I will be generating quite a few time slices of the simulation.
The LCRC Accounts System