December 2014 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 22 Dec '14

22 Dec '14

Hello, A change in allocation has been requested: Requester: heinonen (Olle Heinonen) Project: Meso Title: Mesoscale computational materials science Description: In the first project, mesoscale coupled elastic systems, we are developing new methods and codes (within the MOOSE framework) as well as using the FEniCS code to more finite mesoscale systems in which elastic degrees of freedom are coupled to others, such as concentration or bound and free electric charges. Both the MOOSE and FEniCS codes use MPI and threading to parallelize and both have exhibited strong scaling up towards 100k cores. In these simulations we may use up to 512 cores for production runs. These codes both use PETSc, and MOOSE also uses LibMesh. We compile our own versions of these libraries. In the second project, oxygen defects, we are using a variety of first-principle electronic structure codes, principally SIESTA and FHI-aims, to calculate electronic properties and oxygen diffusion barriers (using nudged elastic band methods). For the heterostructures, we use the non-equilibrium transport code SMEAGOL to calculate the electronic transport. Our developer SIESTA/SMEAGOL version uses threading and MPI for parallelization and we have demonstrated scaling of up to ~64k cores. The FHI-aims code uses MPI only and is a very efficient codes. On blues, we expect to run up to 512 cores for the calculations; for much bigger systems, we are using Mira. We also have a collaboration with Subramanian Sankanayaranyan (CNM) who does MD simulations of interfaces that we feed into the electronic structure calculations. The third project, magnetization behavior, uses a home-grown code that uses OpenMP threading only, so this one runs only on up to 16 cores at a time per job but typically runs at close to 95% efficiency on 16 cores. The number of project members is five to six. Current: undetermined amount Justification: The main reason we need a larger (500k) allocation is that we have several rather heavy users in the project. The codes we use are FEnICs (a PETSc-based FEM code that scales very well), a development version of SIESTA/smeagol with proven scaling of up to ~16k cores, FHI-aims which is very efficient with its ELPA libarary, and a very efficient single-node multi-thread micromagnetic codes. Requested: 500000 A specific reason has been given: Because of heavy use we have already exhausted an initial 210k core-hour allocation for FY2014. We run efficient codes but have 3 - 4 heavy users and need a larger allocation. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for LowF_VRC-TST
by support＠lcrc.anl.gov 15 Dec '14

15 Dec '14

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2014/?project=LowF_VRC-TST

1 0

New LCRC report submitted for LowF_VRC-TST
by support＠lcrc.anl.gov 15 Dec '14

15 Dec '14

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2014/?project=LowF_VRC-TST

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 10 Dec '14

10 Dec '14

Hello, A change in allocation has been requested: Requester: aithal (Shashi Aithal) Project: Nek5000 Title: Scalable CFD Algorithms Description: Nek5000 is a petascale fluid/thermal simulation code used for a host of applications including reactor thermal hydraulics, combustion, MHD, oceanography, spatiotemporal chaos, and vascular flow modeling. Nek5000 is a grid-based PDE (partial differential equation) solver in three space dimensions and time. The code has long been at the forefront of high-order methods and scalable algorithms. It was one of the first comprehensive software tools developed for distributed memory parallel platforms. It pioneered the development of high-order discretizations that, because of their ability to accurately capture interactions over a broad range of scales, are now deemed an essential ingredient of petascale simulation science and that are being adapted across virtually all disciplines (e.g., fusion, aeronautics, biofluids and geophysics). We note that we have access to a dedicated 128-core linux cluster (under a nuclear engineering project) and to resources on BG/P through INCITE and discretionary allocations. This request is to address mid-range capabilities -- particularly for those applications that would not fit on the 128-core cluster as discussed below. Our goal is to use Blues to test the performance of mid-size problems (with 10,000 to 100000) elements with various compilers and compiler options. We also intend to set up a suite of large problems to ensure that various builds are working correctly. We also intend to run production cases for problems in nuclear engineering and conjugate heat-transfer. This year we intend to test various large-element cases which have been run with LES models using 3 different uRANS models that have been implemented in Nek5000. These runs typically need about 100-150K core-hours per case. We aim to study 4-6 such cases with the allocation this year. Current: undetermined amount Justification: Requested: 200000 A specific reason has been given: This additional allocation request is for time till March 2014. A large DNS simulation was run for an APS conference which used up the previously allocated time. Smaller RANS simulations are planned with the additional time till March 2015 This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 08 Dec '14

08 Dec '14

Hello, A change in allocation has been requested: Requester: heinonen (Olle Heinonen) Project: Meso Title: Mesoscale computational materials science Description: In the first project, mesoscale coupled elastic systems, we are developing new methods and codes (within the MOOSE framework) as well as using the FEniCS code to more finite mesoscale systems in which elastic degrees of freedom are coupled to others, such as concentration or bound and free electric charges. Both the MOOSE and FEniCS codes use MPI and threading to parallelize and both have exhibited strong scaling up towards 100k cores. In these simulations we may use up to 512 cores for production runs. These codes both use PETSc, and MOOSE also uses LibMesh. We compile our own versions of these libraries. In the second project, oxygen defects, we are using a variety of first-principle electronic structure codes, principally SIESTA and FHI-aims, to calculate electronic properties and oxygen diffusion barriers (using nudged elastic band methods). For the heterostructures, we use the non-equilibrium transport code SMEAGOL to calculate the electronic transport. Our developer SIESTA/SMEAGOL version uses threading and MPI for parallelization and we have demonstrated scaling of up to ~64k cores. The FHI-aims code uses MPI only and is a very efficient codes. On blues, we expect to run up to 512 cores for the calculations; for much bigger systems, we are using Mira. We also have a collaboration with Subramanian Sankanayaranyan (CNM) who does MD simulations of interfaces that we feed into the electronic structure calculations. The third project, magnetization behavior, uses a home-grown code that uses OpenMP threading only, so this one runs only on up to 16 cores at a time per job but typically runs at close to 95% efficiency on 16 cores. The number of project members is five to six. Current: undetermined amount Justification: Requested: 220000 A specific reason has been given: We need another 10k core-hours Q1 FY2015 in order to finish up some production runs. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 07 Dec '14

07 Dec '14

Hello, A change in allocation has been requested: Requester: ssom (Sibendu Som) Project: LES-Engine Title: Effect of LES resolved spray coupled with reduced kinetics on engine performance Description: With the use of parallel processing and highly-scalable CFD solvers, LES is becoming an increasingly practical technique for modeling the turbulent flow in internal combustion engines. Recently it has been shown by Argonne and CSI that LES can provide good qualitative and quantitative comparisons to instantaneous engine spray measurements since it directly resolves the large scales in the flow field. Grid-convergence behavior of a dynamic structure based LES model was demonstrated by employing minimum cell sizes in the order of 30 microns under evaporating but non-combustion spray conditions. This methodology has not yet been applied to the simulation of a diesel engine. Demonstration of grid-convergence in a real engine simulation is challenging due the need to resolve finer flow structures across larger volumes which may result in high computational costs. This project will address these challenges by using advanced load-balancing algorithms (such as METIS), reduced chemical kinetic models for combustion (ensuring that the fidelity of the model is very high under the operating conditions), multi-zone model, HPC resources to ensure lower wall-clock times, adaptive mesh resolution technique to add cells in desired locations of high temperature, species, and velocity gradients, and sector simulations for bench-marking the LES. The key will be to develop “best practices” for LES simulations on GE hardware. Through this collaboration the following activities are planned; 1. Dynamic structure based LES turbulence model coupled with reduced kinetics to evaluate engine performance using a sector mesh. a. This coupling of LES turbulence models with reduced kinetics is being attempted for the first time. A sector mesh is planned for to keep run times less than a week. 2. Comparison of existing modeling capability against the advanced approaches highlighted above. If indeed these advanced modeling techniques coupled with supercomputing is beneficial for GE; then we need to assess the scalability and speed-up of CONVERGE code with GE-hardware as a continuing exercise. Current: undetermined amount Justification: Requested: 499000 A specific reason has been given: The GE folks decided to run some really large LES simulations of their large bore engines which was originally out of scope for this year. However, the simulations seem successful. We would need to request a similar amount of additional allocation to still finish the planned simulations for this year. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 07 Dec '14

07 Dec '14

Hello, A change in allocation has been requested: Requester: ssom (Sibendu Som) Project: GasTurbineCombustion Title: Benchmark Combustion Simulations for Gas Turbine Applications Description: With the use of parallel processing and highly-scalable CFD solvers, LES is becoming an increasingly practical technique for modeling the turbulent flow in internal combustion engines. Recently it has been shown by Argonne and CSI that LES can provide good qualitative and quantitative comparisons to instantaneous engine spray measurements since it directly resolves the large scales in the flow field. Grid-convergence behavior of a dynamic structure based LES model was demonstrated by employing minimum cell sizes in the order of 30 microns under evaporating but non-combustion spray conditions. This methodology has not yet been applied to the simulation of a gas turbine combustion using the CONVERGE code. We propose to run the combustion simulations at a lower equivalence ratio with an LES turbulence model. Using the LES model should capture the unsteady combustion behavior. A disadvantage of LES turbulence model is that the simulations typically require more grid resolution than a RANS simulation. Comparisons on emissions (CO, CO2, NOx) will be performed between RANS and LES simulations. Demonstration of grid-convergence in a real engine simulation is challenging due the need to resolve finer flow structures across larger volumes which may result in high computational costs. This project will address these challenges by using advanced load-balancing algorithms (such as METIS), reduced chemical kinetic models for combustion (ensuring that the fidelity of the model is very high under the operating conditions), multi-zone model, HPC resources to ensure lower wall-clock times, adaptive mesh resolution technique to add cells in desired locations of high temperature, species, and velocity gradients, and sector simulations for bench-marking the LES. The key will be to develop “best practices” for LES gas turbine combustion simulations. Current: undetermined amount Justification: Requested: 350000 A specific reason has been given: This project is in collaboration with CSI for benchmarking large flame simulations simulations. We underestimated the wall-clock time for these simulations originally. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 07 Dec '14

07 Dec '14

Hello, A change in allocation has been requested: Requester: ssom (Sibendu Som) Project: gm_gasexchange Title: Large Eddy Simulation of IC Engine Flows Description: CFD with RANS (Reynolds-Averaged Navier-Stokes) approach has been applied to engine development for many decades and are still the main tool for engine design today. Due to solving ensemble or phase-averaged mean quantities, RANS is very much limited for average cycles, hence cannot capture cycle-to-cycle variations. The high fidelity DNS (direct numerical simulation) is too computationally intensive and is still not feasible for full engine simulations in the near future. By solving spatially filtered Navier-Stokes equation, LES has been highlighted to have the potential for predicting the details of the transient in-cylinder flows and offer the essential information regarding the nature of cycle-to-cycle variations a decade ago. However, the computational requirements and need for multiple-cycle simulations required by LES hindered its further research and application for IC engines. Together with great leap of computer power in past decade, Argonne’s and G M R&D’s expertise in LES calculations, and availability of high-fidelity experimental data from University of Michigan (funded by GM R&D), this proposal aims at establishing the best engineering practices for LES calculations in IC engines. The increased predictive capability with LES will enable more efficient and robust engine design. GM R&D has developed many engineering CFD practices for RANS and LES simulations. One example is the numerical setup of valve opening and closing events for in-cylinder flow simulations. In the past, pending on the meshing topology used, a minimum valve lift together with a crank angle position is arbitrarily chosen to mimic the valve opening and closing events. In most cases, the minimum valve lift is determined mainly from solution stability and simulation cost considerations. This is true especially when the meshing topology involves non-orthogonal skewed meshes. A lack of accurate flow field measurements for validation in the valve vicinity also hinders progress in this area. Recent meshing advancements in CONVERGE in conjunction with the availability of accurate high speed PIV data render a new opportunity for such study. Using these toolsets, GM R&D has established a strategy for accurate predictions of engine in-cylinder flows for RANS simulations. The next step is to test this RANS methodology for LES simulations. With this proposal, the following main components of LES for engine flows will be employed: (1) Use dynamic structure sub-grid scale (SGS) turbulence model on unresolved small flow scales; (2) Use the best CFD setup derived from RANS simulation on minimum valve lift, opening /closing crank angle position and minimum mesh size in the valve seat region; (3) Use a 2nd-order convective scheme; (4) Evaluate model accuracy against available measured data from University of Michigan; (5) Explore different data analysis methods like POD on LES results. In summary, this proposal will be executed in a manner such that CFD simulations will collaborate closely with experimental diagnostics. The high-speed PIV data for the TCC-III engine from University of Michigan will be available for LES model validation. The ultimate goal will be to use LES for IC engine simulations for in-cylinder processes involving engine flow, spray, combustion, and soot emissions. The specific methods used for these flow simulations with moving boundaries will include: 1) Finite volume schemes 2) RANS turbulence models 3) CONVERGE software is being used - 2 base and unlimited child licenses available Current: undetermined amount Justification: The simulations will be performed with CONVERGE code. As reported by all the other projects of the PI (Sibendu Som), the scaling efficiency is quite good (more than 70%) up to 2000 processors for typical engine problems Requested: 200000 A specific reason has been given: We are running several LES cycles. After running several cycles we found that the set-up was incorrect. We would need to rerun the whole set-up again and hence requesting more core hours. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 04 Dec '14

04 Dec '14

Hello, A change in allocation has been requested: Requester: svegendla (Prasad Vegendla) Project: NE_RANS Title: RANS-based CFD for Nuclear Reactors Description: Anticipated CFD support of nuclear reactor analysis in FY15 includes: • Supporting NNSA’s efforts under the GTRI-Convert program to convert research reactors fueled with highly-enriched uranium to instead employ low-enriched uranium, and in doing so reduce the associated proliferation risk. Such reactor conversions require significant analysis to ensure that their cool ability under a range of conditions. Given the range of geometry and flow conditions presented by the various research reactors under consideration, a general-purpose thermofluid analysis capability—like CFD—is required. • Supporting advanced reactor systems modeling by coupling STAR-CCM+ to system codes for transient analysis of postulated adverse events, to ensure that the reactor remains cool even under highly unlikely multi-component failures. • Supporting advanced modeling and simulation efforts under the DOE-NE NEAMS program by integrating STAR-CCM+ into the SHARP reactor analysis package. By including STAR-CCM+ as an optional component in SHARP will provided needed flexibility and enhance the access for new users to the advanced modeling and simulation toolkit. • Support of design and safety analysis of sodium-cooled fast reactors for the DOE-NE Advanced Reactor Concepts program. • Verification and validation of STAR-CCM+ for nuclear rector analysis. • In work for others program, several STAR-CCM+ simulation runs are planned to optimize the aerodynamic and underhood thermal analysis of the heavy-duty vehicles. STAR-CCM+ has been employed by analysts in the Nuclear Engineering Division on LCRC clusters since its inception (both Jazz and Fusion), and has demonstrated good scalability up to several hundred cores. Recent simulations on 720 Fusion cores reported only 15% overhead associated with MPI communication. This project will be used by 5-10 STAR-CCM+ analysts in the NE Division. Current: undetermined amount Justification: STARCCM+ is a highly scalable tool on fusion. NE users demonstrated good scalability up to several hundred cores. Recent simulations on 720 Fusion cores reported only 15% overhead associated with MPI communication. Requested: 360000 A specific reason has been given: *The proposed amount mostly used for StarCCM+ Vehicle aerodrag simulations, under the influence of traffic conditions. *These simulations runs are performed upto 40 to 120 Mi volume cells. *A Total of 30 simulation runs are planned. *Each run will take approximately 12000 hrs to converge. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 04 Dec '14

04 Dec '14

Hello, A change in allocation has been requested: Requester: svegendla (Prasad Vegendla) Project: CFD-TPM Title: Computational Fluid Dynamics Modeling of Two-phase Boiling Flow and Critical Heat Flux Description: In boiling water reactors, dryout is an important limiting phenomenon in the design of heat exchangers and is associated with the disappearance of the liquid film on the wall in heat exchanger tube. A consequence of this non-wetting heated surface deteriorates the convective heat transfer mechanism resulting in a large rise in the wall temperature of heat flux imposed system such as nuclear reactor fuel rod assemblies. Since dryout is a hydrodynamic phenomenon involving entrainment and redeposition of droplets as well as boiling/evaporation of the liquid, the location of dryout is dynamic and its fluctuation may also give rise to thermal fatigue and corrosion. A large number of experimental and theoretical studies have been conducted over the past several decades to understand and quantify the processes leading to dryout in a heat exchanger tube. * Nek5000 is a highly-scalable open-source transient CFD code developed by MCS-Argonne National Laboratory. This code is based on spectral element method and it is written in basic FORTRAN and C languages. It has more than 200 users world-wide and a long history of development, verification, and validation. * In this project, Nek5000 is used as a platform to develop drift-flux and two-fluid model. * In order to validate the Nek5000 development for two-phase boiling capability, several simulation runs are planned for wide range of pressure, mass flux and wall heat flux. * In FY2014, blues allocated hours are efficiently used for projects of NE_RANS and SHARP-IF. * A total of 3 members can use the allocated hours Current: undetermined amount Justification: Nek5000 is a highly scalable code and it is used for wide range of applications. Requested: 400000 A specific reason has been given: Nek5000 boiling flow code testing and validation takes considerably heavy computational time. Additional hours are needed to do approximately 20 more runs, each run will take around 20000 hrs. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0