February 2014 - Allocations-admins - mailman3.cels.anl.gov

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Valdes 18 Feb '14

18 Feb '14

On Tue, Feb 18, 2014 at 04:08:22PM +0000, Bair, Raymond A. wrote: > Since time is still available, lets approve this. Approved (cfd_enginemodeling). John

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Valdes 18 Feb '14

18 Feb '14

> Lets provide enough to support the 80 nodes for the rest of the quarter. OK, I've granted ApsRenewalLattice more time. John

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 18 Feb '14

18 Feb '14

Hello, A change in allocation has been requested: Requester: heinonen (Olle Heinonen) Project: Meso Title: Mesoscale computational materials science Description: This project has three components.1 1. Magnetization configurations and dynamics in nanoscale magnetic systems uses two codes, one is an F90 OpenMP threaded code (no MPI parallellization) using MKL libraries. This code runs very well and scales well up to 48 cores (max I have tried). It runs efficiently on 8 cores and "perfect scaling" will be obtained by running on multiple nodes simultaneously. I am also in the process of acquiring and FEM-MPI codes through collaborators. This code is a C code using MPI parallelization and standard libraries, and has been shown to scale well on up to ~100 cores. Again, multiple runs will typically be done in parallel. 2. Coupled elastic-ferroic systems is mostly development work using the MOOSE framework. MOOSE is built on Libmesh and PETSc that have proven scalability on up to ~10^5 cores. During the year we will also do some production work and scaling studies on the contributions we are making to MOOSE. 3. We will also use development versions of SIESTA-SMEAGOL and MOOSE for coupled QM and elastic systems. The particular version of SIESTA-SMEAGOL has been modified for better scaling. Most of the work here will be development work - we have allocations on BG/P and BG/Q and also an ALCC proposal pending for a major allocation request. The project will have up to 7 members (~3 staff, 2 post-docs and 2 grad students) Current: undetermined amount Justification: Requested: 400000 A specific reason has been given: I ran out of the current allocation. I have been running an OpenMP code that runs very efficiently on 16 cores - I need more runs of that code. Also, we are ramping up a project using a developer version of SIESTA/SMEAGOL that will need many more nodes/cores for pre-production runs before we move to BG/Q This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 18 Feb '14

18 Feb '14

Since time is still available, lets approve this. Ray ----------------------------------------- Ray Bair Computing, Environment, and Life Sciences Argonne National Laboratory and the University of Chicago TCS Building 240, Room 4122 9700 South Cass Avenue Argonne, IL 60439 email: rbair(at)anl.gov Phone: (630)252-5751 On 2/18/14 8:54 AM, "accounts(a)lcrc.anl.gov" <accounts(a)lcrc.anl.gov> wrote: >Hello, > >A change in allocation has been requested: > > Requester: ssom (Sibendu Som) > Project: cfd_enginemodeling > Title: Computation Fluid Dynamics Modeling of Diesel Engine >Processes > Description: The project is divided into three major phases: >1) Cavitation modeling: A two-phase cavitation modeling approach using >the homogeneous relaxation method has been developed and implemented in >CONVERGE. Extensive validation of the new model will be performed against >experimental data available in literature and new data from Advanced >photon source at Argonne. >2) Spray Modeling: Traditional Eulerian-Lagrangian spray modeling >approaches are hampered by the fact that the results are not >grid-independent. Grid-convergent spray modeling approaches will be >developed and validated against experimental data from the Engine >Combustion Network. The simulations will be performed in a constant >volume combustion vessel since it is a more effective crucible for model >validation >3) Turbulence Modeling: Traditionally engine simulations have been >performed using the Reynolds Average Navier Stokes Equations (RANS) >approach. RANS employs filtering in time to derive the governing >equations for the mean state. Turbulent interactions over the full range >of dynamic scales are averaged to make the calculations affordable for >engineering analysis. Only the largest energy containing features in a >flow are resolved and no information exists to describe broadband >small-scale interactions. Thus, computational requirements for RANS are >relatively low and the most affordable. However, case-by-case calibration >of models is required, and stochastic processes such as cycle-to-cycle >variations cannot be captured in RANS. Large Eddy Simulation (LES) >directly resolves the large scale unsteady motions that account for the >bulk spatial transportation, while smaller scales of the flow are removed >through a filtering operation and treated by using subgrid scale (SGS) >models f > or reduced computational cost. By resolving the large-scale turbulence, >LES can significantly improve the accuracy of flow predictions compared >to RANS. A dynamic structure based LES model will be developed and the >results will be compared against RANS. >The CONVERGE software has been observed to scale well up to 256 >processors and full-cycle engine simulations. > > Current: undetermined amount >Justification: > > Requested: 100000 > >A specific reason has been given: >Need to run more LES simulations > >This needs to be approved and the final allocation amount decided upon. > >Thank You, >The LCRC Accounts System >_______________________________________________ >allocations-admins mailing list >allocations-admins(a)lists.lcrc.anl.gov >https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 18 Feb '14

18 Feb '14

Hello, A change in allocation has been requested: Requester: ssom (Sibendu Som) Project: cfd_enginemodeling Title: Computation Fluid Dynamics Modeling of Diesel Engine Processes Description: The project is divided into three major phases: 1) Cavitation modeling: A two-phase cavitation modeling approach using the homogeneous relaxation method has been developed and implemented in CONVERGE. Extensive validation of the new model will be performed against experimental data available in literature and new data from Advanced photon source at Argonne. 2) Spray Modeling: Traditional Eulerian-Lagrangian spray modeling approaches are hampered by the fact that the results are not grid-independent. Grid-convergent spray modeling approaches will be developed and validated against experimental data from the Engine Combustion Network. The simulations will be performed in a constant volume combustion vessel since it is a more effective crucible for model validation 3) Turbulence Modeling: Traditionally engine simulations have been performed using the Reynolds Average Navier Stokes Equations (RANS) approach. RANS employs filtering in time to derive the governing equations for the mean state. Turbulent interactions over the full range of dynamic scales are averaged to make the calculations affordable for engineering analysis. Only the largest energy containing features in a flow are resolved and no information exists to describe broadband small-scale interactions. Thus, computational requirements for RANS are relatively low and the most affordable. However, case-by-case calibration of models is required, and stochastic processes such as cycle-to-cycle variations cannot be captured in RANS. Large Eddy Simulation (LES) directly resolves the large scale unsteady motions that account for the bulk spatial transportation, while smaller scales of the flow are removed through a filtering operation and treated by using subgrid scale (SGS) models f or reduced computational cost. By resolving the large-scale turbulence, LES can significantly improve the accuracy of flow predictions compared to RANS. A dynamic structure based LES model will be developed and the results will be compared against RANS. The CONVERGE software has been observed to scale well up to 256 processors and full-cycle engine simulations. Current: undetermined amount Justification: Requested: 100000 A specific reason has been given: Need to run more LES simulations This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 17 Feb '14

17 Feb '14

Lets provide enough to support the 80 nodes for the rest of the quarter. Ray ----------------------------------------- Ray Bair Computing, Environment, and Life Sciences Argonne National Laboratory and the University of Chicago TCS Building 240, Room 4122 9700 South Cass Avenue Argonne, IL 60439 email: rbair(at)anl.gov Phone: (630)252-5751 On 2/17/14 5:19 PM, "John Valdes" <valdes(a)anl.gov> wrote: >Michael's project has run out of time, so he can't currently submit >jobs (at least not for his desired runtime). He's asking for another >2 million for the remainder of the quarter. Note that 80 nodes for 43 >days (approx number of days remaining in the quarter) comes to ~1.32M >corehours. > >Do we approve? > >John > >On Mon, Feb 17, 2014 at 04:54:40PM -0600, accounts(a)lcrc.anl.gov wrote: >> >> A change in allocation has been requested: >> >> Requester: borland (Michael Borland) >> Project: ApsRenewalLattice >[...] >> Requested: 2000000 >> >> A specific reason has been given: >> On-going development of the lattice for APS upgrade needs significant >>resources and will continue to do so for about another month as we >>respond to the recent physics review and prepare for the Conceptual >>Design Report. >_______________________________________________ >allocations-admins mailing list >allocations-admins(a)lists.lcrc.anl.gov >https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Valdes 17 Feb '14

17 Feb '14

Michael's project has run out of time, so he can't currently submit jobs (at least not for his desired runtime). He's asking for another 2 million for the remainder of the quarter. Note that 80 nodes for 43 days (approx number of days remaining in the quarter) comes to ~1.32M corehours. Do we approve? John On Mon, Feb 17, 2014 at 04:54:40PM -0600, accounts(a)lcrc.anl.gov wrote: > > A change in allocation has been requested: > > Requester: borland (Michael Borland) > Project: ApsRenewalLattice [...] > Requested: 2000000 > > A specific reason has been given: > On-going development of the lattice for APS upgrade needs significant resources and will continue to do so for about another month as we respond to the recent physics review and prepare for the Conceptual Design Report.

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 17 Feb '14

17 Feb '14

Hello, A change in allocation has been requested: Requester: borland (Michael Borland) Project: ApsRenewalLattice Title: Lattice Development and Beam Dynamics for APS Upgrade Description: APS has developed a method of using multi-objective genetic algorithms (MOGA) for optimization of accelerator lattices. In past years, this method has been applied successfully using fusion, intrepid, and a small APS cluster to development and evaluation of lattices for the APS Upgrade. In the coming year, we are working on a multi-bend achromat (MBA) lattice design that promises to push APS x-ray brightness to world-leading levels. Significant progress has already been made and results are very encouraging. Our approach is to create a series of lattices that gradually push performance up while simultaneously incorporating on-going constraints derived from on-going interactions with magnet and vacuum system designers. As such, the process is naturally iterative, working toward a major technical review in December 2013 and a major project review in Spring 2014. In addition to MOGA, we will perform ensemble evaluation to assess the robustness of solutions, evaluate effects of insertion devices, and explore collective instabilities. The codes being used have all been used in past years on fusion. These include Pelegant, a parallel accelerator simulation code used primarily for particle tracking and acceptance determination, as well as a Tcl script used for genetic optimization. The simulations are close to embarrassingly parallel and thus scale very well. Current: undetermined amount Justification: The code is already running efficiently on Blues. Scaling is very good because the code is close to embarrassingly parallel. The MOGA script runs many simultaneous, unconnected jobs, each using 2 nodes. Because each job uses only 32 cores, scaling is not relevant. Jobs are sized to take advantage of the number of cores provided, e.g., by tracking n*32 particles so that each core has an approxiamtely equal workload. Requested: 2000000 A specific reason has been given: On-going development of the lattice for APS upgrade needs significant resources and will continue to do so for about another month as we respond to the recent physics review and prepare for the Conceptual Design Report. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 17 Feb '14

17 Feb '14

Hello, A change in allocation has been requested: Requester: sarich (Jason Sarich) Project: UNEDF_Optimization Title: Optimization for the NUCLEI SciDAC-3 Description: We are building upon our optimization successes under the SciDAC-2 UNEDF (Universal Nuclear Energy Density Functional) project in the successor project NUCLEI. In this project, optimizations over parameter spaces corresponding to energy density functional parameters will be performed using the Argonne-developed POUNDERS optimization algorithm. We will work with DFT codes HFBTHO and HFODD and begin work with ab initio codes such as MFDn. These physics codes will be run using parallel resources, in some cases with each of 100 nuclei being run on a separate node, in others (such as with MFDn), a simulating a single nucleus will require up to hundreds of nodes. In the past we have used OpenMP at the nucleus level in concert with library and compiler-based optimizations to bring the wall-clock time of a single by a factor 10. We will continue such code optimizations for the next-generation codes developed under NUCLEI. We anticipate using 40k core-hours for this task in FY14. For our parameter optimizations, we typically have 10-20 parameters and require on the order of 300 simulations. Based on the current simulation time, we would thus require 120 8-core nodes for up to 6 hours, or 6k core-hours for each version of the functional we want to optimize. We expect the number of core-hours will remain the same when using the new 16-core nodes. Past experience has let us to expect to do roughly 40 such optimizations in FY13, for a total of 240k core-hours for this task. POUNDERS is available in the Toolkit of Advanced Optimization (TAO), but we still prototype specialized versions of the algorithm using MATLAB/Octave on a single, master node. We will test updates to the TAO version on Blues and compare the performance and correctness against the MATLAB/Octave versions. As in the past, we will have roughly 4 users for this project. Current: undetermined amount Justification: N/a Requested: 120000 A specific reason has been given: We have a paper deadline approaching soon for the NUCLEI project, we need to get several more optimization runs to get it finished. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 07 Feb '14

07 Feb '14

Hello, A change in allocation has been requested: Requester: ssom (Sibendu Som) Project: LES-Engine Title: Effect of LES resolved spray coupled with reduced kinetics on engine performance Description: With the use of parallel processing and highly-scalable CFD solvers, LES is becoming an increasingly practical technique for modeling the turbulent flow in internal combustion engines. Recently it has been shown by Argonne and CSI that LES can provide good qualitative and quantitative comparisons to instantaneous engine spray measurements since it directly resolves the large scales in the flow field. Grid-convergence behavior of a dynamic structure based LES model was demonstrated by employing minimum cell sizes in the order of 30 microns under evaporating but non-combustion spray conditions. This methodology has not yet been applied to the simulation of a diesel engine. Demonstration of grid-convergence in a real engine simulation is challenging due the need to resolve finer flow structures across larger volumes which may result in high computational costs. This project will address these challenges by using advanced load-balancing algorithms (such as METIS), reduced chemical kinetic models for combustion (ensuring that the fidelity of the model is very high under the operating conditions), multi-zone model, HPC resources to ensure lower wall-clock times, adaptive mesh resolution technique to add cells in desired locations of high temperature, species, and velocity gradients, and sector simulations for bench-marking the LES. The key will be to develop “best practices” for LES simulations on GE hardware. Through this collaboration the following activities are planned; 1. Dynamic structure based LES turbulence model coupled with reduced kinetics to evaluate engine performance using a sector mesh. a. This coupling of LES turbulence models with reduced kinetics is being attempted for the first time. A sector mesh is planned for to keep run times less than a week. 2. Comparison of existing modeling capability against the advanced approaches highlighted above. If indeed these advanced modeling techniques coupled with supercomputing is beneficial for GE; then we need to assess the scalability and speed-up of CONVERGE code with GE-hardware as a continuing exercise. The specific methods used for the multi-phase, reacting flow simulations with moving boundaries will include: 1) Finite volume schemes 2) LES (dynamic structure) and RANS turbulence models used 3) Detailed chemical kinetic models for diesel fuels 4) CONVERGE software is being used - ~128 licenses available 5) Scales well up-to 128 processors 6) Detailed spray models used for different spray processes Current: undetermined amount Justification: Requested: 75000 A specific reason has been given: We would like to redistribution the allocation given to this project from 450K, 50K, 0K and 0K for Q1, Q2, Q3 and Q4, respectively, to 125K per quarter. Towards that end, since we've been granted 50K for the current quarter, we are requesting an addition 75K for the current quarter. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0