Stas and I added this and gave no time on Blues.
Thanks!
-John
From: allocations-admins <allocations-admins-bounces@lcrc.anl.gov> on behalf of "Bair, Raymond A." <rbair@anl.gov>
Reply-To: LCRC Allocations Admins <allocations-admins@lcrc.anl.gov>
Date: Friday, September 1, 2017 at 11:48 AM
To: LCRC Allocations Admins <allocations-admins@lcrc.anl.gov>
Subject: [allocations-admins] Request for interim 3DDES project allocation of 1M core hours on Bebop
The allocations committee seems OK with 3DDES request for 1M Bebop core-hours in Sept. Please make the allocation, with the understanding that Ramesh will spend most of the time on KNL.
Regards,
Ray
-----------------------------------------
Ray Bair
Argonne National Laboratory
and the University of Chicago
From: "Wagner, Albert F." <wagner@anl.gov>
Date: Thursday, August 31, 2017 at 3:26 PM
To: Raymond Bair <rbair@anl.gov>
Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop
Ray,
I am OK with this.
Al
From: alloc-com <alloc-com-bounces@lists.lcrc.anl.gov> on behalf of "Bair, Raymond A." <rbair@anl.gov>
Date: Thursday, August 31, 2017 at 3:21 PM
To: "alloc-com@lcrc.anl.gov" <alloc-com@lcrc.anl.gov>
Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop
Are there any additional comments on this request? Since he can spend most of the 1M core-hours time on KNL nodes, which are ore lightly used at this time, it should not load down Bebop.
Regards,
Ray
-----------------------------------------
Ray Bair
Argonne National Laboratory
and the University of Chicago
From: Raymond Bair <rbair@anl.gov>
Date: Wednesday, August 30, 2017 at 12:30 PM
To: "Pieper, Steven C." <spieper@anl.gov>, "alloc-com@lcrc.anl.gov" <alloc-com@lcrc.anl.gov>
Cc: Raymond Bair <rbair@anl.gov>
Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop
Ramesh writes . . .
The current set of jobs that I have ready to go (in fact, I was running them during the EU period), have meshes partitioned for 4096 cores. So, they can be run either on 128x32 cores on (BDW) or 64x64 cores (on KNL) although the KNL would
take a longer wall clock time. For my runs, I can do a 25% (BDW) 75%(KNL) split, since there are ways to accelerate the solution on the KNL nodes.
-----------------------------------------
Ray Bair
Argonne National Laboratory
and the University of Chicago
From: "Pieper, Steven C." <spieper@anl.gov>
Organization: Physics, Argonne National Laboratory
Reply-To: "Pieper, Steven C." <spieper@anl.gov>
Date: Wednesday, August 30, 2017 at 11:08 AM
To: Raymond Bair <rbair@anl.gov>
Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop
I guess OK although it is ~1/16 of the total Broadwell hours
available for Sept.
Steve
Steven C. Pieper, Senior Physicist Emeritus: spieper@anl.gov
Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439
Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100
http://www.phy.anl.gov/theory/staff/scp.html
On 08/30/17 11:02, Bair, Raymond A. wrote:
LCRC Allocations Committee,
Ramesh Balakrishnan would like to continue his early user work on Bebop, transitioning into a larger project in FY2018 (project 3DDES). For the remainder of FY2017, her requests 1m core-hours. See his email below.
Clearly the time is available on Bebop, and you will act on the full FY2018 proposal (also below) at your September meeting. Do you concur with providing the 1M core-hour interim allocation? Please respond by Thursday, August 31.
Regards,
Ray
-----------------------------------------
Ray Bair
Argonne National Laboratory
and the University of Chicago
From: Argonne Service Desk <argonne@service-now.com>
Reply-To: "argonne@service-now.com"<argonne@service-now.com>
Date: Monday, August 28, 2017 at 3:56 PM
To: Raymond Bair <rbair@anl.gov>, "Aithal, Shashikant M." <aithal@cels.anl.gov>
Subject: Request AREQ0175370 Watch List: Request for an interim 3DDES project allocation of 1M core hours on Bebop from (August 28, September 30], 2017 Queue: HPC (CE
You can reply directly to this email. Please enter your response above this line.
Hello,
You have been added to the watch list for this request and will receive updates as work on the ticket progresses.
If you have additional information that may help in fulfilling this request, please reply to this message,
Number: AREQ0175370
Customer: Ramesh Balakrishnan
Short Description: Request for an interim 3DDES project allocation of 1M core hours on Bebop from (August 28, September 30], 2017 Queue: HPC (CELS)
Details:
received from: bramesh@alcf.anl.gov
Hello All,
Based on my e-discussions with Shashi Aithal, I would like to request an allocation of 1M core hours on Bebop to continue my Early User work till September 30, 2017. I have submitted a project request for an allocation that I understand will start from October 1, 2017 (if the proposal is accepted). If possible, please grant me 1 TB of project space for this short term allocation of 1M core hours for the same project name, i.e. 3DDES. Do let me know if you have any questions.
Thanks,
Ramesh
Thank you,
Argonne Service Desk
630-252-9999, option 2
help@anl.gov
Ref:MSG1993806
THE FOLOLWING IS THE FY2018 REQEUST.
Hello,
A new project on the LCRC cluster has been requested. Please forward
the information on to the LCRC Allocation sub-committee.
Applicant's name: Ramesh Balakrishnan
Applicant's institution: ANL
Applicant's division: LCF
Project Name: 3DDES
Project title: Detached Eddy Simulation of Separated High Reynolds Number Flows
Associated funding: ASCR Funding via ALCF
Other Systems: Theta, Mira, Bebop
Science: I am requesting 10M core hours, for the whole year, on Bebop, to develop detached eddy simulation models that can serve as a wall model for moderately separated flow in thick boundary layers. Examples of such flow are the atmospheric boundary layer (ABL) over complex terrain, large eddy breakup devices (LEBU), and separated flow past airfoils at high angles of attack (i.e. greater than 10 degrees).
Project description: Preliminary simulations of the ABL over Complex Terrain have been carried out on Bebop during the Early User Period, where the PI had an opportunity to run and test out modified version(s) of OpenFOAM, SU2, and FDL3DI on Bebop. The simulations were run on meshes ranging from 28M grid points to 140M grid points, and on a maximum of 4096 cores of Bebop. The codes make use of MPI and can optionally use OpenMP. In addition, one can vectorize the code for the KNL nodes (in which case OpenMP is not needed at all). During the early user period, the vectorization and OpenMP options were not tested out. The flat MPI codes do scale well on both the KNL and Broadwell nodes, although the time-to-solution is about five to six times slower on the KNL nodes (when compared to the Broadwell nodes). This is something that I plan to investigate in more detail as part of this allocation request.
Industry partnership: None at this time.
Project URL:
Requested allocation: 10000000
Q1: 2500000
Q2: 2500000
Q3: 2500000
Q4: 2500000
Justification: modified OpenFOAM: 35,000 finite volumes per core (on smallest number of cores)
=======================================
cores time/time_step
4096 (KNL) 27 seconds (64 cores/node x 64 nodes)
2048 (KNL) 43 seconds (64 cores/node x 32 nodes)
1024 (KNL) 75 seconds (64 cores/node x 16 nodes)
4096 (BDW) 2 seconds (32 cores/node x 128 nodes)
2048 (BDW) 5 seconds (32 cores/node x 64 nodes)
Please note that the 10M core hours (for the whole year) is requested on Bebop.
1024 (BDW) 12 seconds (32 cores/node x 32 nodes)
modified SU2: 12,000 finite volumes per core (on smallest # of cores)
=================================
solver cores time/time_step
FV 1024 (KNL) 11 seconds
FE_DG (p4) 1024 (KNL) 16 seconds (same # finite volumes/core)
FV 2048 (KNL) 8 seconds
FE_DG (p4) 2048 (KNL) 11 seconds (same # finite volumes per core)
Storage requirements: 5TB
I will be generating quite a few time slices of the simulation.
_______________________________________________alloc-com mailing listalloc-com@lists.lcrc.anl.govhttps://lists.lcrc.anl.gov/mailman/listinfo/alloc-com