- Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Yearly Allocation Request for PROTEUS_2015
by accounts＠lcrc.anl.gov 01 Sep '17

01 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Emily Shemon Project Name: PROTEUS_2015 Division: NE Project title: PROTEUS Neutronics Analysis Associated funding: DOE NEAMS Program Other Systems: Small NE Linux clusters, Mira (BG/Q) [ALCC award] Science: Continuing from FY17, DOE NEAMS is funding work to perform hot channel factor calculations as well and mixed fidelity "zooming" calculations with the SHARP toolkit (PROTEUS, Nek) for nuclear reactor geometries, specifically the AFR-100 design. Project description: The two codes used will be PROTEUS-MOCEX (hot channel factor) and PROTEUS-SN (zooming). Both are high fidelity neutron transport solvers and use pure MPI distributed memory parallelism. The scaling limits of PROTEUS-SN is well known (90%+ efficiency when using 2 angles/process and >1500 vertices/process). The scaling limits of PROTEUS-MOCEX are less well known but some lower limits have been established with previous runs. The codes compile and are routinely tested with Intel compilers. External software packages are necessary (PETSc, HDF5, Metis) some of which will be installed by the team, some of which are already available on the system. Specifically, we need to run several (30+) 91-pin assembly calculations with PROTEUS-MOCEX where each calculation contains a randomized distribution of fuel pellet enrichment. The results will be post-processed and used to determine a variance on the resulting distribution, which can then be used in the Nek5000 code to translate this variance into temperature and hot channel factors. Two temperatures distributions may need to be tested which doubles the count to 60 simulations. Some testing and debugging is expected. PROTEUS-MOCEX will most likely be used - it easily performs on the intermediate-sized problems proposed that fit well with the Blues size and time constraints. We have an allocation on Mira that will be necessary for our very largest simulations, but many of our calculations are intermediate size (too large for local cluster, too small for Mira) and perfect size for LCRC. We have run MOCEX successfully on Blues with 16 core per node for the exact problem of interest during FY17. We have not yet run on Bebop, but do not anticipate any severe issues. It is possible we may be slightly more memory limited on Bebop and need to use fewer cores/node than the maximum. On Blues, the finest fidelity PROTEUS-MOCEX simulations required 48 cores for 4.25 wall clock hours (~200 core-hours per run). The lowest fidelity used 48 cores for 1.25 wall clock hours. The parallelization was tailored to the Blues nodes (16 cores per node). We will most likely use either 1-2 Broadwell nodes, or 1 KNL node. If memory per core is insufficient on Bebop, we will need to use more nodes and have some inactive cores on each node. 60 runs of these yields an estimate of 15000 core-hours. However, some testing may be needed to refine the mesh, etc, and also examine different temperature distributions so we increase that estimate to 20000 core hours. The PROTEUS-SN code is much more expensive, requiring 2048 cores for 4.25 hours (9000 core-hours per run) for the above problem. We do not plan to use PROTEUS-SN for the runs already described. However, due to larger geometry, PROTEUS-SN may be required for the full core simulations (zooming capability) which are much larger than the 91-pin single assembly calculations. We plan to run our full production simulations on Blue Gene/Q however some testing will be needed at smaller scale which doesn't fit well with the BG/Q 60 min time limit (at smaller scale). We expect 50,000 core-hours per simulation, of which we need to perform at least 3-4 over the year. Additionally, the Nek5000 code may need to run small calculations so we added in some extra time below to account for that. This NEAMS work is hoped to be completed by March with a shift to other tasks at that point. So we weight the first two quarters heaviest in our request (150K, 100K) with baseline funding for future work (100K, 100K) for the last two quarters. Industry partnership: n/a Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 450000 Q1: 150000 Q2: 100000 Q3: 100000 Q4: 100000 Justification: n/a Storage requirements: 1 TB is fine Thank You, The LCRC Accounts System

1 0

radix request for more time
by Bair, Raymond A. 01 Sep '17

01 Sep '17

Misun talked with Pavan, and he is willing to try to stay within 100K additional hours. So, please add that to project radix. Regards, Ray ----------------------------------------- Ray Bair Computing, Environment, and Life Sciences Argonne National Laboratory and the University of Chicago TCS Building 240, Room 4122 9700 South Cass Avenue Argonne, IL 60439 email: rbair(at)anl.gov Phone: (630)252-5751

1 0

Re: [allocations-admins] Request for interim 3DDES project allocation of 1M core hours on Bebop
by Roberts, John E. 01 Sep '17

01 Sep '17

Stas and I added this and gave no time on Blues. Thanks! -John From: allocations-admins <allocations-admins-bounces(a)lcrc.anl.gov> on behalf of "Bair, Raymond A." <rbair(a)anl.gov> Reply-To: LCRC Allocations Admins <allocations-admins(a)lcrc.anl.gov> Date: Friday, September 1, 2017 at 11:48 AM To: LCRC Allocations Admins <allocations-admins(a)lcrc.anl.gov> Subject: [allocations-admins] Request for interim 3DDES project allocation of 1M core hours on Bebop The allocations committee seems OK with 3DDES request for 1M Bebop core-hours in Sept. Please make the allocation, with the understanding that Ramesh will spend most of the time on KNL. Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: "Wagner, Albert F." <wagner(a)anl.gov> Date: Thursday, August 31, 2017 at 3:26 PM To: Raymond Bair <rbair(a)anl.gov> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop Ray, I am OK with this. Al From: alloc-com <alloc-com-bounces(a)lists.lcrc.anl.gov<mailto:[email protected]>> on behalf of "Bair, Raymond A." <rbair(a)anl.gov<mailto:[email protected]>> Date: Thursday, August 31, 2017 at 3:21 PM To: "alloc-com(a)lcrc.anl.gov<mailto:[email protected]>" <alloc-com(a)lcrc.anl.gov<mailto:[email protected]>> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop Are there any additional comments on this request? Since he can spend most of the 1M core-hours time on KNL nodes, which are ore lightly used at this time, it should not load down Bebop. Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: Raymond Bair <rbair(a)anl.gov<mailto:[email protected]>> Date: Wednesday, August 30, 2017 at 12:30 PM To: "Pieper, Steven C." <spieper(a)anl.gov<mailto:[email protected]>>, "alloc-com(a)lcrc.anl.gov<mailto:[email protected]>" <alloc-com(a)lcrc.anl.gov<mailto:[email protected]>> Cc: Raymond Bair <rbair(a)anl.gov<mailto:[email protected]>> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop Ramesh writes . . . The current set of jobs that I have ready to go (in fact, I was running them during the EU period), have meshes partitioned for 4096 cores. So, they can be run either on 128x32 cores on (BDW) or 64x64 cores (on KNL) although the KNL would take a longer wall clock time. For my runs, I can do a 25% (BDW) 75%(KNL) split, since there are ways to accelerate the solution on the KNL nodes. ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: "Pieper, Steven C." <spieper(a)anl.gov<mailto:[email protected]>> Organization: Physics, Argonne National Laboratory Reply-To: "Pieper, Steven C." <spieper(a)anl.gov<mailto:[email protected]>> Date: Wednesday, August 30, 2017 at 11:08 AM To: Raymond Bair <rbair(a)anl.gov<mailto:[email protected]>> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop I guess OK although it is ~1/16 of the total Broadwell hours available for Sept. Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov<mailto:[email protected]> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/30/17 11:02, Bair, Raymond A. wrote: LCRC Allocations Committee, Ramesh Balakrishnan would like to continue his early user work on Bebop, transitioning into a larger project in FY2018 (project 3DDES). For the remainder of FY2017, her requests 1m core-hours. See his email below. Clearly the time is available on Bebop, and you will act on the full FY2018 proposal (also below) at your September meeting. Do you concur with providing the 1M core-hour interim allocation? Please respond by Thursday, August 31. Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: Argonne Service Desk <argonne(a)service-now.com><mailto:[email protected]> Reply-To: "argonne(a)service-now.com"<mailto:[email protected]><argonne(a)service-now.com><mailto:[email protected]> Date: Monday, August 28, 2017 at 3:56 PM To: Raymond Bair <rbair(a)anl.gov><mailto:[email protected]>, "Aithal, Shashikant M." <aithal(a)cels.anl.gov><mailto:[email protected]> Subject: Request AREQ0175370 Watch List: Request for an interim 3DDES project allocation of 1M core hours on Bebop from (August 28, September 30], 2017 Queue: HPC (CE You can reply directly to this email. Please enter your response above this line. ________________________________ Hello, You have been added to the watch list for this request and will receive updates as work on the ticket progresses. If you have additional information that may help in fulfilling this request, please reply to this message, Number: AREQ0175370<https://argonne.service-now.com/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=…> Customer: Ramesh Balakrishnan Short Description: Request for an interim 3DDES project allocation of 1M core hours on Bebop from (August 28, September 30], 2017 Queue: HPC (CELS) Details: received from: bramesh(a)alcf.anl.gov<mailto:[email protected]> Hello All, Based on my e-discussions with Shashi Aithal, I would like to request an allocation of 1M core hours on Bebop to continue my Early User work till September 30, 2017. I have submitted a project request for an allocation that I understand will start from October 1, 2017 (if the proposal is accepted). If possible, please grant me 1 TB of project space for this short term allocation of 1M core hours for the same project name, i.e. 3DDES. Do let me know if you have any questions. Thanks, Ramesh Thank you, Argonne Service Desk 630-252-9999, option 2 help(a)anl.gov<mailto:[email protected]> Ref:MSG1993806 THE FOLOLWING IS THE FY2018 REQEUST. Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Ramesh Balakrishnan Applicant's institution: ANL Applicant's division: LCF Project Name: 3DDES Project title: Detached Eddy Simulation of Separated High Reynolds Number Flows Associated funding: ASCR Funding via ALCF Other Systems: Theta, Mira, Bebop Science: I am requesting 10M core hours, for the whole year, on Bebop, to develop detached eddy simulation models that can serve as a wall model for moderately separated flow in thick boundary layers. Examples of such flow are the atmospheric boundary layer (ABL) over complex terrain, large eddy breakup devices (LEBU), and separated flow past airfoils at high angles of attack (i.e. greater than 10 degrees). Project description: Preliminary simulations of the ABL over Complex Terrain have been carried out on Bebop during the Early User Period, where the PI had an opportunity to run and test out modified version(s) of OpenFOAM, SU2, and FDL3DI on Bebop. The simulations were run on meshes ranging from 28M grid points to 140M grid points, and on a maximum of 4096 cores of Bebop. The codes make use of MPI and can optionally use OpenMP. In addition, one can vectorize the code for the KNL nodes (in which case OpenMP is not needed at all). During the early user period, the vectorization and OpenMP options were not tested out. The flat MPI codes do scale well on both the KNL and Broadwell nodes, although the time-to-solution is about five to six times slower on the KNL nodes (when compared to the Broadwell nodes). This is something that I plan to investigate in more detail as part of this allocation request. Industry partnership: None at this time. Project URL: Requested allocation: 10000000 Q1: 2500000 Q2: 2500000 Q3: 2500000 Q4: 2500000 Justification: modified OpenFOAM: 35,000 finite volumes per core (on smallest number of cores) ======================================= cores time/time_step 4096 (KNL) 27 seconds (64 cores/node x 64 nodes) 2048 (KNL) 43 seconds (64 cores/node x 32 nodes) 1024 (KNL) 75 seconds (64 cores/node x 16 nodes) 4096 (BDW) 2 seconds (32 cores/node x 128 nodes) 2048 (BDW) 5 seconds (32 cores/node x 64 nodes) Please note that the 10M core hours (for the whole year) is requested on Bebop. 1024 (BDW) 12 seconds (32 cores/node x 32 nodes) modified SU2: 12,000 finite volumes per core (on smallest # of cores) ================================= solver cores time/time_step FV 1024 (KNL) 11 seconds FE_DG (p4) 1024 (KNL) 16 seconds (same # finite volumes/core) FV 2048 (KNL) 8 seconds FE_DG (p4) 2048 (KNL) 11 seconds (same # finite volumes per core) Storage requirements: 5TB I will be generating quite a few time slices of the simulation. _______________________________________________ alloc-com mailing list alloc-com(a)lists.lcrc.anl.gov<mailto:[email protected]> https://lists.lcrc.anl.gov/mailman/listinfo/alloc-com

1 0

[LCRC Accounts] Yearly Allocation Request for SHARP_ARC
by accounts＠lcrc.anl.gov 01 Sep '17

01 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Yiqi Yu Project Name: SHARP_ARC Division: NE, MCS Project title: Multi-physics Simulation of SFR Core Radial Expansion with SHARP Associated funding: DOE-NE Advanced Reactor Concepts Other Systems: NE Division clusters, MCS Cosmea cluster Science: Following multi-year efforts for development of coupled neutronics, computational fluid dynamics (CFD), and computational structural mechanics (CSM) analysis capabilities, the SHARP multi-physics nuclear reactor analysis toolkit was released for distribution (via an end-user license agreement) at the end of March 2016. In FY17, our main focus will therefore shift toward testing, validation and verification of SHARP component codes and multi-physics toolkit in application to challenge problems for advanced reactor designs. The objective is to assess SHARP capabilities in addressing the high-fidelity modeling and simulation needs that have been identified as important to customers in the nuclear industry and other DOE R&D programs (particularly the Advanced Reactor Technologies program). One challenge problem that has been identified is the need to assess hot pin/hot channel factors for sodium-cooled fast reactor designs. Hot pin and hot channel factors are a means of accounting for deviations between the actual manufactured nuclear fuel pins and the design specifications, as well as deviations in the actual operating flow conditions from the idealized flow field assumed for design calculations. In this effort, SHARP will be used to explicitly model the standard assumptions made in conservative analyses by reassessing hot-channel and hot-pin factors for Argonne’s AFR-100 reactor concept. The fuel assembly with peak power-to-flow ratio, and the peak-to-average pin power, temperature and flow distributions within that fuel assembly under nominal conditions will be identified. The impact of inlet flow maldistribution and resulting temperature variations, cladding circumferential temperature variation, uncertainties and tolerances in fuel manufacturing, core lo ading errors, and uncertainties in heat transfer correlations will also be assessed. If successful, reliable prediction of these factors will enhance the safety margins and therefore improve the economic competitiveness of new reactor concepts. A second challenge problem is to demonstrate “SHARP Zoom”, a multi-scale modeling capability for a focused view of a select region of an otherwise homogenized core model. The “SHARP Zoom” concept is a multi-scale modeling technique aimed to achieve a high-fidelity (focused) view of a select zone (i.e., fuel assembly) as part of a homogenized whole-core analysis to reduce the computational requirements and cost significantly in comparison to a full-core pin-by-pin analysis. Further, the allocation will be used to support international collaborations for code validation activities. The first is a U.S.-Japan Bilateral collaboration on sodium-cooled fast reactor technology development. For the advanced modeling & simulation task under the U.S.-Japan Bilateral Commission on Civil Nuclear Cooperation, benchmark simulation of a sodium experiment using wire-wrapped 37-pin bundles from JAEA’s Plant Dynamic Test Loop facility will be pursued to validate SHARP CFD solvers for liquid metal coolants. Under a collaboration between the U.S. and European Union, there are tasks earmarked for further SHARP CFD validation, including exercises related to flow in wire-wrapped fuel assemblies, and validation of approaches that couple conventional 1-D codes with 3-D CFD. Project description: As the SHARP modeling and simulation toolkit is maturing, the focus is on customer-oriented applications and further validation exercises. In Fiscal year 2018, the SHARP tool will be applied to hot channel factor simulation for sodium fast reactor. It will help SAHRP to demonstrate the capability of helping the new design for nuclear reactor and save some safety margin. This allocation will help with those applications, for performing small to moderate scale simulations and for scaling up to ALCF. The SHARP toolkit is regularly compiled on Blues using Intel or GNU compilers. The components of SHARP—the Nek5000 computational fluid dynamics code and the PROTEUS neutron transport code—have already been demonstrated to scale well not only on Blues but on Blue Gene/P. This project will be used by 4-5 analysts in the NE and MCS Divisions. Industry partnership: Project URL: http://energy.gov/ne/advanced-modeling-simulation/advanced-nuclear-reactors Current FY Hours Used: undetermined amount New FY Requested allocation: 490000 Q1: 100000 Q2: 140000 Q3: 100000 Q4: 150000 Justification: Storage requirements: 4 TB would provide convenience, considering the relatively large number of collaborators working simultaneously. Thank You, The LCRC Accounts System

1 0

radix request for 500K Blues time in Sept
by Bair, Raymond A. 01 Sep '17

01 Sep '17

Update: Some people on the Allocations Committee feel that Blues is too busy to add 500K to radix in Sept., so Misun is going to talk with Pavan. Stay tuned. Regards, Ray

1 0

Request for interim 3DDES project allocation of 1M core hours on Bebop
by Bair, Raymond A. 01 Sep '17

01 Sep '17

The allocations committee seems OK with 3DDES request for 1M Bebop core-hours in Sept. Please make the allocation, with the understanding that Ramesh will spend most of the time on KNL. Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: "Wagner, Albert F." <wagner(a)anl.gov> Date: Thursday, August 31, 2017 at 3:26 PM To: Raymond Bair <rbair(a)anl.gov> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop Ray, I am OK with this. Al From: alloc-com <alloc-com-bounces(a)lists.lcrc.anl.gov<mailto:[email protected]>> on behalf of "Bair, Raymond A." <rbair(a)anl.gov<mailto:[email protected]>> Date: Thursday, August 31, 2017 at 3:21 PM To: "alloc-com(a)lcrc.anl.gov<mailto:[email protected]>" <alloc-com(a)lcrc.anl.gov<mailto:[email protected]>> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop Are there any additional comments on this request? Since he can spend most of the 1M core-hours time on KNL nodes, which are ore lightly used at this time, it should not load down Bebop. Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: Raymond Bair <rbair(a)anl.gov<mailto:[email protected]>> Date: Wednesday, August 30, 2017 at 12:30 PM To: "Pieper, Steven C." <spieper(a)anl.gov<mailto:[email protected]>>, "alloc-com(a)lcrc.anl.gov<mailto:[email protected]>" <alloc-com(a)lcrc.anl.gov<mailto:[email protected]>> Cc: Raymond Bair <rbair(a)anl.gov<mailto:[email protected]>> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop Ramesh writes . . . The current set of jobs that I have ready to go (in fact, I was running them during the EU period), have meshes partitioned for 4096 cores. So, they can be run either on 128x32 cores on (BDW) or 64x64 cores (on KNL) although the KNL would take a longer wall clock time. For my runs, I can do a 25% (BDW) 75%(KNL) split, since there are ways to accelerate the solution on the KNL nodes. ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: "Pieper, Steven C." <spieper(a)anl.gov<mailto:[email protected]>> Organization: Physics, Argonne National Laboratory Reply-To: "Pieper, Steven C." <spieper(a)anl.gov<mailto:[email protected]>> Date: Wednesday, August 30, 2017 at 11:08 AM To: Raymond Bair <rbair(a)anl.gov<mailto:[email protected]>> Subject: Re: [LCRC alloc-com] Request for interim 3DDES project allocation of 1M core hours on Bebop I guess OK although it is ~1/16 of the total Broadwell hours available for Sept. Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov<mailto:[email protected]> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/30/17 11:02, Bair, Raymond A. wrote: LCRC Allocations Committee, Ramesh Balakrishnan would like to continue his early user work on Bebop, transitioning into a larger project in FY2018 (project 3DDES). For the remainder of FY2017, her requests 1m core-hours. See his email below. Clearly the time is available on Bebop, and you will act on the full FY2018 proposal (also below) at your September meeting. Do you concur with providing the 1M core-hour interim allocation? Please respond by Thursday, August 31. Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago From: Argonne Service Desk <argonne(a)service-now.com><mailto:[email protected]> Reply-To: "argonne(a)service-now.com"<mailto:[email protected]><argonne(a)service-now.com><mailto:[email protected]> Date: Monday, August 28, 2017 at 3:56 PM To: Raymond Bair <rbair(a)anl.gov><mailto:[email protected]>, "Aithal, Shashikant M." <aithal(a)cels.anl.gov><mailto:[email protected]> Subject: Request AREQ0175370 Watch List: Request for an interim 3DDES project allocation of 1M core hours on Bebop from (August 28, September 30], 2017 Queue: HPC (CE You can reply directly to this email. Please enter your response above this line. ________________________________ Hello, You have been added to the watch list for this request and will receive updates as work on the ticket progresses. If you have additional information that may help in fulfilling this request, please reply to this message, Number: AREQ0175370<https://argonne.service-now.com/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=…> Customer: Ramesh Balakrishnan Short Description: Request for an interim 3DDES project allocation of 1M core hours on Bebop from (August 28, September 30], 2017 Queue: HPC (CELS) Details: received from: bramesh(a)alcf.anl.gov<mailto:[email protected]> Hello All, Based on my e-discussions with Shashi Aithal, I would like to request an allocation of 1M core hours on Bebop to continue my Early User work till September 30, 2017. I have submitted a project request for an allocation that I understand will start from October 1, 2017 (if the proposal is accepted). If possible, please grant me 1 TB of project space for this short term allocation of 1M core hours for the same project name, i.e. 3DDES. Do let me know if you have any questions. Thanks, Ramesh Thank you, Argonne Service Desk 630-252-9999, option 2 help(a)anl.gov<mailto:[email protected]> Ref:MSG1993806 THE FOLOLWING IS THE FY2018 REQEUST. Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Ramesh Balakrishnan Applicant's institution: ANL Applicant's division: LCF Project Name: 3DDES Project title: Detached Eddy Simulation of Separated High Reynolds Number Flows Associated funding: ASCR Funding via ALCF Other Systems: Theta, Mira, Bebop Science: I am requesting 10M core hours, for the whole year, on Bebop, to develop detached eddy simulation models that can serve as a wall model for moderately separated flow in thick boundary layers. Examples of such flow are the atmospheric boundary layer (ABL) over complex terrain, large eddy breakup devices (LEBU), and separated flow past airfoils at high angles of attack (i.e. greater than 10 degrees). Project description: Preliminary simulations of the ABL over Complex Terrain have been carried out on Bebop during the Early User Period, where the PI had an opportunity to run and test out modified version(s) of OpenFOAM, SU2, and FDL3DI on Bebop. The simulations were run on meshes ranging from 28M grid points to 140M grid points, and on a maximum of 4096 cores of Bebop. The codes make use of MPI and can optionally use OpenMP. In addition, one can vectorize the code for the KNL nodes (in which case OpenMP is not needed at all). During the early user period, the vectorization and OpenMP options were not tested out. The flat MPI codes do scale well on both the KNL and Broadwell nodes, although the time-to-solution is about five to six times slower on the KNL nodes (when compared to the Broadwell nodes). This is something that I plan to investigate in more detail as part of this allocation request. Industry partnership: None at this time. Project URL: Requested allocation: 10000000 Q1: 2500000 Q2: 2500000 Q3: 2500000 Q4: 2500000 Justification: modified OpenFOAM: 35,000 finite volumes per core (on smallest number of cores) ======================================= cores time/time_step 4096 (KNL) 27 seconds (64 cores/node x 64 nodes) 2048 (KNL) 43 seconds (64 cores/node x 32 nodes) 1024 (KNL) 75 seconds (64 cores/node x 16 nodes) 4096 (BDW) 2 seconds (32 cores/node x 128 nodes) 2048 (BDW) 5 seconds (32 cores/node x 64 nodes) Please note that the 10M core hours (for the whole year) is requested on Bebop. 1024 (BDW) 12 seconds (32 cores/node x 32 nodes) modified SU2: 12,000 finite volumes per core (on smallest # of cores) ================================= solver cores time/time_step FV 1024 (KNL) 11 seconds FE_DG (p4) 1024 (KNL) 16 seconds (same # finite volumes/core) FV 2048 (KNL) 8 seconds FE_DG (p4) 2048 (KNL) 11 seconds (same # finite volumes per core) Storage requirements: 5TB I will be generating quite a few time slices of the simulation. _______________________________________________ alloc-com mailing list alloc-com(a)lists.lcrc.anl.gov<mailto:[email protected]> https://lists.lcrc.anl.gov/mailman/listinfo/alloc-com

1 0

Re: [allocations-admins] [lcrc-core] Outstanding Allocations Questions 8/30/17
by Bair, Raymond A. 01 Sep '17

01 Sep '17

There’s another new project request today, that did not go through the renewal link: DFT_UMo, 150K Q4, Zhigang Mei ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 8/31/17, 3:19 PM, "Roberts, John E." <jeroberts(a)anl.gov> wrote: IP-ML wants 50K for Q4 and 250K through FY18. Pinging the rest. Thanks! -John On 8/30/17, 11:11 AM, "lcrc-core on behalf of Bair, Raymond A." <lcrc-core-bounces(a)lists.lcrc.anl.gov on behalf of rbair(a)anl.gov> wrote: LCRC, Since I am traveling today, here are what I see as the open issues for LCRC Allocations this week. Radix, request to add 500K. Ray sent to Allocations Committee for decision. 3DDES, early user/new project, requests interim 1M in Q4 on Bebop. Ray sent to Allocations Committee for decision. See new project requests below. These did not come in as “year allocation”. Do they want FY17 start? Or, do we need to move these to the FY18 allocation requests? Please check with PIs. NEMphyValidation, 80K IP-ML, 250K BEM_Uncertainty, 55K in Q4 REMM, 483K in Q4 ATLAS-HEP-group, 100K in Q4 Regards, Ray _______________________________________________ lcrc-core mailing list lcrc-core(a)lists.lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/lcrc-core

1 0

New LCRC report submitted for uo2_md
by support＠lcrc.anl.gov 31 Aug '17

31 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=uo2_md

1 0

New LCRC report submitted for lith_ox_md
by support＠lcrc.anl.gov 31 Aug '17

31 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=lith_ox_md

1 0

New LCRC report submitted for SnI4
by support＠lcrc.anl.gov 31 Aug '17

31 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=SnI4

1 0