September 2011 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Yearly Allocation Request from Bubble
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Zeke Insepov Project Name: Bubble Division: MCS, NE Project title: Multi-scale simulation of bubbles in nuclear fuels Associated funding: RERTR, AFCI, Argonne LDRD Other Systems: Blue Gene Science: Summary of the project: Emission of energetic fission gases heavily damage nuclear fuels that leads to radiation cascade generation, amorphization, defect formation, lattice swelling, cracking and eventually to failure. Our simulation will be directed toward understanding these and other phenomena. Chemical and physical studies of cavity and vapor-gas filled bubble formation in metals have been conducted with continuum mechanics formulations in conjunction with empirically-derived equations of state. The condensation-evaporation phenomena at liquid-gas interface, dissociation and ionization phenomena which can occur during super compression and super heating for complex molecules has not been possible to assess on a molecular dynamics scale. DFT Ab-initio and classical molecular dynamics simulations will be performed for a number of points of the phase diagram region corresponding to the transient conditions the liquid-bubble interface experiences during the super-compression cycles. The formation of defects in bcc Mo lattice as a result of 50-keV Xe ion bombardment will be studied via atomistic simulation with an interatomic potential developed using the force-matching Ab initio based approach. Radiation defect formation via nuclear stopping power and evolution of the defects in the cascade will be described. Diffusion and interaction of interstitials and vacancies will be analyzed. Project description: computational methods: DFT siesta and vasp codes, classical Molecular Dynamics and kinetic Monte Carlo simulations will used programming model: parallel experiments planned: Argonne IVEM and Atlas experiments software requirements: C++, fortran compiler (e.g. openmpi) sizes of calculations run on and/or planned for LCRC or other systems: hundreds of nano-seconds time scale MD simulations on 1e6-1e8 atomic systems expected number of project members: 7 We expect a total of 7 project members (two from Purdue University, one from the University of Illinois, and two from the New University of Astana). Results of assessments are expected to lead to joint proposals to Argonne LDRD Strategic initiative, NSF, DOE (Office of Science, and NE), and DARPA, two applications were submitted to obtain International funding. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 1000000 Q1: 250000 Q2: 250000 Q3: 250000 Q4: 250000 Justification: Scalability and Computational Efficiency for the simulations using VASP and LAMMPS codes were investigated. Good scaling was seen up to 8 nodes for the problems studied by VASP and up to 64 nodes for LAMMPS simulations. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A change in allocation has been requested: Requester: insepov (Zeke Insepov) Project: xenon Title: Multi-scale simulation of radiation damage in nuclear fuels Description: The following calculations will be performed for interatomic potentials Mo-Mo, Mo-Xe, U-U in the form of EAM potential: 1. Choosing a structure fitting the potential, calculations of energies and forces on atoms in this configurations from DFT by VASP. 2. Creating the potential by force-matching software PotFit. 3. Verification of the potential obtained by comparison with the DFT data: formation energies of vacancies, self-interstitial atoms (SIA) and their clusters, energies of the basic structures of SIAs, their hierarchy. 4. Verification of the potential with the experimental data on diffusion of defects (diffusion constants, threshold temperatures to induce long-range migration) and production of defects in radiation cascades (structure of the defects and sizes). 5.Improvement of the reference structures to reduce the discrepancies revealed in 3 and 4. computational methods: Density Functional Molecular Dynamics simulations using the siesta method programming model: parallel experiments planned: Argonne IVEM and Atlas experiments, Electron Microscopy software requirements: C++, fortran compiler (e.g. openmpi) sizes of calculations run on and/or planned for LCRC or other systems: hundreds of nano-seconds time scale for classical MD simulations on 1e7-1e8 atomic systems expected number of project members: 7 We expect a total of 7 project members (two from University of Illinois, and several members from the New University of Astana and Russian Academy of Sciences. Results of assessments are expected to lead to joint proposals to NSF, DOE (Office of Science, and NE), and DARPA. Current: undetermined amount Justification: We recently purchased a new software - vasp 4.6 and upgraded it to 5.2. Thay will be widely applied to nuclear fuel simulation. All the software involved demonstrates good parallel performance and scalability for the tasks described above. So the typical number of CPUs varies from tens to hundreds (VASP, PotFit) or thousands (LAMMPS). Requested: 840000 A specific reason has been given: We applied for funding and developed a new collaboration with the New University team on various nuclear and plasma simulation projects. Therefore, we expect new members. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A change in allocation has been requested: Requester: insepov (Zeke Insepov) Project: xenon Title: Multi-scale simulation of radiation damage in nuclear fuels Description: The following calculations will be performed for interatomic potentials Mo-Mo, Mo-Xe, U-U in the form of EAM potential: 1. Choosing a structure fitting the potential, calculations of energies and forces on atoms in this configurations from DFT by VASP. 2. Creating the potential by force-matching software PotFit. 3. Verification of the potential obtained by comparison with the DFT data: formation energies of vacancies, self-interstitial atoms (SIA) and their clusters, energies of the basic structures of SIAs, their hierarchy. 4. Verification of the potential with the experimental data on diffusion of defects (diffusion constants, threshold temperatures to induce long-range migration) and production of defects in radiation cascades (structure of the defects and sizes). 5.Improvement of the reference structures to reduce the discrepancies revealed in 3 and 4. computational methods: Density Functional Molecular Dynamics simulations using the siesta method programming model: parallel experiments planned: Argonne IVEM and Atlas experiments, Electron Microscopy software requirements: C++, fortran compiler (e.g. openmpi) sizes of calculations run on and/or planned for LCRC or other systems: hundreds of nano-seconds time scale for classical MD simulations on 1e7-1e8 atomic systems expected number of project members: 7 We expect a total of 7 project members (two from University of Illinois, and several members from the New University of Astana and Russian Academy of Sciences. Results of assessments are expected to lead to joint proposals to NSF, DOE (Office of Science, and NE), and DARPA. Current: undetermined amount Justification: We recently purchased a new software - vasp 4.6 and upgraded it to 5.2. Thay will be widely applied to nuclear fuel simulation. All the software involved demonstrates good parallel performance and scalability for the tasks described above. So the typical number of CPUs varies from tens to hundreds (VASP, PotFit) or thousands (LAMMPS). Requested: 840000 A specific reason has been given: We applied for funding and developed a new collaboration with the New University team on various nuclear and plasma simulation projects. Therefore, we expect new members. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A change in allocation has been requested: Requester: insepov (Zeke Insepov) Project: Bubble Title: Multi-scale simulation of bubbles in nuclear fuels Description: computational methods: DFT siesta and vasp codes, classical Molecular Dynamics and kinetic Monte Carlo simulations will used programming model: parallel experiments planned: Argonne IVEM and Atlas experiments software requirements: C++, fortran compiler (e.g. openmpi) sizes of calculations run on and/or planned for LCRC or other systems: hundreds of nano-seconds time scale MD simulations on 1e6-1e8 atomic systems expected number of project members: 7 We expect a total of 7 project members (two from Purdue University, one from the University of Illinois, and two from the New University of Astana). Results of assessments are expected to lead to joint proposals to Argonne LDRD Strategic initiative, NSF, DOE (Office of Science, and NE), and DARPA. Current: undetermined amount Justification: We recently purchased a new software (vasp 4.6) and upgraded it to vasp 5.2. They will be applied to nuclear fuel simulation. One of the applications of the vasp software is the new interatomic potential for the Molybdenum-Xenon and Uranium-Molybdenum-Xenon systems that have been recognized as a breakthrough achievement for the future metal fuel simulations. Requested: 1000000 A specific reason has been given: We expect several new members for this project from our new collaboration with Kazakhstan. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from aimd_catalysis
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: John J. Low Project Name: aimd_catalysis Division: MCS Project title: Ab Initio Molecular Dynamics for Catalysis of Biomass to Fuels Associated funding: Department of Energy, Office of Science, and Office of Basic Energy Sciences Other Systems: None Science: Since the catalytic conversion of biomass occurs in the aqueous phases, water will be intimately involved with the reaction mechanisms. Ab Initio Molecular Dynamics allows for the quantum mechanical simulation of reaction mechanisms in aqueous phases under reaction conditions such as low pH and high temperature (~500K). Metadynamics accelerates the molecular dynamics along collective variables and enables the mapping of the free energy surfaces from reactants to products with trajectories covering several picoseconds. These trajectories reveal the reaction mechanism and estimate the free energies of activation for competing pathways. This project will model the acid catalyzed decomposition of glucose to hydroxy-methyl furfural (HMF) in water. HMF is precursor to transportation fuels and industrial chemicals. The AIMD of reactions with explicit water molecules is more realistic than the self consistent reaction fields used to predict thermodynamics in solutions with quantum chemistry because clusters of water hydrogen bond strongly to glucose and mediates hydrogen transfer reactions in aqueous phases. The results of the AIMD will be used to model the effects of mixed solvents on the glucose decomposition reaction and improved yields of fuel from glucose. Project description: We will use CPMD to model reactions in aqueous phases. This program has the very efficient Carr-Parinello algorithm for AIMD and several different methods for thermodynamics sampling including metadynamics, umbrella sampling and the Blue-Moon ensemble. CPMD is an excellent tool for modeling reactions in liquids and sampling phase space. CPMD has been installed on Fusion for general use. Several Argonne researchers have been trained to use this software in their research. A workshop will further promote the use of CPMD to model reactions. The parallel performance of CPMD is fair. The unit cell for our model of glucose in sulfuric acid contains 230 atoms and a unit cell volume of 2500 Å3. The MPI version of CPMD had a parallel efficiency of 70% on 160 processors for this model. The number of planes in the real space grid limits the number of processors which can be used efficiently by CPMD. A metadynamics run of 10ps is adequate to model a step in the glucose decomposition reaction. Ten picoseconds require 7.3 kilocore-hours. We plan to model ten reaction pathways in the glucose to HMF reaction mechanism in three different solvent mixtures. This work will require approximately 200 kilocore-hours. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 200 Q1: 50 Q2: 50 Q3: 50 Q4: 50 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from qmc_for_nuclei
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Robert Wiringa Project Name: qmc_for_nuclei Division: PHY Project title: Quantum Monte Carlo Calculations of Light Nuclei Associated funding: DOE Division of Nuclear Physics Other Systems: ANL Blue Gene/P (INCITE 15M hours for 12C studies) ANL Blue Gene/Q (early user time for same) ANL SiCortex (indeterminate) Science: This project uses Quantum Monte Carlo [Green's function (GFMC), variational (VMC), and auxiliary-field diffusion (AFDMC)] methods to compute ground-state and low-lying excited-state expectation values of energies, densities, structure functions, astrophysical reaction rates, etc., for light nuclei and low-energy scattering reactions involving these nuclei. Realistic two- and three-nucleon potentials are used. Our goal is a description of all of these systems using a Hamiltonian that also provides an excellent description of nucleon-nucleon scattering and nucleonic matter. Such a `standard nuclear model' can then be used, for example, to compute low-energy astrophysical reactions which cannot be experimentally measured. Project description: In FY11 we made fairly full use of our allotted Fusion time - about 160,000 cpu hours total. This is in addition to extensive use of the MCS SiCortex and ALCF Blue Gene for major parts of our research. We expect to continue using those other resources but the SiCortex time is not guaranteed. In FY12 there will be three staff members, one postdoc, and one or two external collaborators working on the project. We would like to do the following: A) Clustered wave functions for A=9-12 nuclei We have been making new variational Monte Carlo (VMC) wave functions for A=9-11 nuclei that reflect the inherent clustering in these nuclei in a fashion different from standard shell-model wave functions. These wave functions treat the natural and unnatural-parity states in these nuclei on a more equal footing and are making our first calculations of 11B and 11C possible. There are about eight states each in 10B and 10Be that need to be diagonalized and evaluated, as well as development of the A=11 nuclei, and we estimate a need for 100,000 cpu hours for this program in FY12. B) GFMC Nucleon-Nucleus Scattering We are continuing studies of nucleon-nucleus scattering for n+3H, p+3He, n+4He, and p+4He, and for parity-violating (PV) n+4He scattering using Green's function Monte Carlo (GFMC). There are important unresolved issues with the 3+1 and 4+1 scattering cases, particularly handling coupled channels (for 3+1) and absolute normalizations (for PV 4+1). Based on current experience with Fusion, we will need about 30,000 cpu hours to complete the present studies. C) Spectroscopic overlaps, asymptotic normalizations and widths We have been calculating VMC spectroscopic overlaps, for application to nucleon-knockout experiments and transfer reactions, for many years. We continue to provide new or improved overlaps to experimental groups both at Argonne and elsewhere and wish to continue using our newer cluster-type wave functions described above. In the past year we also developed new integral techniques for evaluating asymptotic normalization constants (ANCs) and single-nucleon widths of narrow particle-unstable states. This work led to three papers (one Physical Review Letter, one Physical Review C Rapid Communication, and one regular Physical Review C article) in the past year. We would like to extend this to alpha-particle ANCs and widths in the next year. For FY12 we estimate 40,000 cpu hours for this part of the project. D) Transitions in A=8-10 Nuclei We have been studying electromagnetic transitions in 10Be and 10C in collaboration with the PHY low-energy experimental group for some time. These studies show a surprising sensitivity to details of the nuclear Hamiltonian, in particular the three-body forces, which we need to explore further. We have recently begun testing the role of charge-symmetry breaking on these transitions, and also examining weak decays in this sector. Our current postdoc has been developing meson-exchange-current contributions to electromagnetic transition operators and we wish to start testing them in the A=8-10 nuclei. For this work we expect to use 40,000 cpu hours in FY12. E) Neutron drops We also continue to use Auxiliary Field Diffusion Monte Carlo (AFDMC) and GFMC to compute properties of large systems of neutrons. These are both neutron drops and neutron matter done in periodic boundary conditions. We have two tasks: 1) to provide guidance to the development of energy density functional theories as part of the UNEDF SciDAC, and 2) to test the effect of three-body forces, fitted to light nuclei, in neutron matter. A Physical Review Letter on this subject was published this year. We estimate 30,000 cpu hours for this work in FY12. This all adds up to 240,000 hours. Project URL: http://www.phy.anl.gov/theory/research/forces.html Current FY Hours Used: undetermined amount New FY Requested allocation: 240000 Q1: 60000 Q2: 60000 Q3: 60000 Q4: 60000 Justification: The Quantum Monte Carlo methods work very efficiently on parallel processors. We use MPI and see speed-up efficiencies better than 95% for up to 250 cpus on Fusion, with sustained speeds of 1.8 GFLOPS/cpu. Using ADLB and OpenMP on BG/P we have run efficiently on 130,000 processors. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from pflotran
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Boyana Norris Project Name: pflotran Division: MCS Project title: PFLOTRAN Performance Analysis and Tuning Associated funding: SciDAC Institute for Sustained Performance, Energy, and Resilience (SUPER) Algorithms and Software for Communication Avoidance and Communication Hiding at the Extreme Scale (CACHE Math/CS Institute) Other Systems: Jazz, Surveyor, Intrepid Science: performance analysis, performance optimizations, and automated empirical tuning Project description: PFLOTRAN is a massively parallel code developed at LANL for the accurate prediction of multiscale subsurface processes. PFLOTRAN has many applications and guides safer industrial techniques; for example, it can help geoscientists determine how carbon dioxide flows through aquifers and deep geologic formations during carbon-capture and storage. In the first years of this project, we focused mainly on analyzing and improving the performance of the sequential portions of key computational kernels (matrix-vector product, triangular linear solution, and some of the chemistry computations), which required many single-core runs of these kernels. During FY12, we will focus on multicore optimizations and extensions of PETSc (used by PFLOTRAN) through our collaboration with RNET Technologies and Ohio State University. As part of the project, the team is investigating data structures, algorithms, compiler optimizations, and novel programming paradigms to enable applications based on the PETSc library to fully utilize the computing power of emerging Petascale compute architectures. While our initial focus is on PFLOTRAN, these modifications facilitate aggressive vectorization and efficient memory accesses and are expected to speed up many PETSc based finite difference and finite volume simulations. The project involves active development and testing of MPI, OpenMP, and manually created or auto-generated architecture intrinsics code. A parallel computing cluster with state-of-the-art architecture is necessary to demonstrate scalability and asymptotic performance of these PETSc modifications. Access to a cluster with multiple nodes, quad core processors, and high-performance interconnects is highly desirable in light of the above efforts. Another new focus for FY12 is dynamic adaptation of the linear solution methods by employing newly developed feature analysis and classification techniques. Based on analysis of limited training data, we identify significant matrix properties, which are then used to select the best solver and preconditioner combination in production runs of PFLOTRAN (and eventually other PETSc based applications). We also plan to continue our work on automatic code generation and tuning for the key linear algebra kernels in PFLOTRAN, which requires empirical testing of many different optimized versions. We will significantly extend the types of optimizations considered, which would correspondingly greatly increase the empirical search space, necessitating advances in the numerical optimization algorithms used to guide the search. We will use PFLOTRAN input decks that can run on different ranges of cores, from one node to the whole machine. We'll target run times of less than an hour in most cases (by changing the number of time evolution steps). Project URL: http://trac.mcs.anl.gov/projects/performance/wiki/OrioPFLOTRAN Current FY Hours Used: undetermined amount New FY Requested allocation: 499900 Q1: 124900 Q2: 125000 Q3: 125000 Q4: 125000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: STability_Nek
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Elia Merzari Applicant's institution: ANL Applicant's division: NE, MCS Project Name: STability_Nek Project title: Development of SPectral element stability analisys tool Associated funding: LDRD Other Systems: Science: To develop a global linear stability analisys code using the Nek5000 framework. The tool suggested can be applied to the analysis of virtually any flow system. Applications involving laminar flows would benefit in particular (e.g., microfluidics, biological flows, buoyancy driven flows) since the identification a priori of the Laminar-Turbulent transition threshold would allow for important design considerations (e.g., in the laminar regime heat transfer is dramatically reduced and this should be taken into account). However the main application of the proposed methodology would be to the identification of geometrical elements that generate unstable modes. This capability would useful for both turbulent and laminar flows. Coupled with the non-linear solver already present in Nek500 to verify the results, this can lead to : 1. the removal of expensive instabilities, that are often associated to thermal striping for instance, 2. the control of turbulent processes; 3. the control of buoyancy-driven phenomena, which are prone to instabilities. The work would be mainly numerical and theoretical in nature, with the development, verification and validation of the methodology on some relevant test cases. Once the methodology in place we plan to test it on some typical benchmark case such as the solution of the stability problem in square channels in 2D and 3D, analyzing the requirement on the spectral discretization. The 2D stability problem for square channels will be delivered at the end of the first year. At the same time several 2D test runs will be attempted. Project description: The Navier-stokes equation for incompressible fluids (with the Boussinesq approximation for the buoyancy term) and the energy equation would be solved through Nek5000 to obtain a steady state. Then the Navier-Stokes equations would be linearized by substituting to each variable the steady state and a superposed Ansatz. By neglecting the higher order terms, the Navier stokes equations can be reduced to a generalized complex eigenvalue problem. The eigenvalues of the linearized operator wuold then be computed through an Arnoldi algorithm. The size of the eigenvalue is large since it scales with (N^2), N being the size of the mesh. Calculations will be largely parallel, only a small portions of calculations will be non-parallel, corresponding to approximately 5%, due mostly to testing. Parallel eigenvalue solvers will be imported/implemented/developed in Nek5000. Nek5000 has excellent scalability proven on both Fusion and BG/P. The following tasks have been identified: 1) development of the 2D methodology; (50,000-100,000 core-hours). 2) test with 2D channel cases (100,000-200,000 core-hours) and 2D cavity problems. 3) test preliminary 3D cases (100,000-200,000 core-hours). The use of the LCRC cluster wuold be essential to achieve the goals of the project. Project URL: Requested allocation: 400000 Q1: 40000 Q2: 160000 Q3: 100000 Q4: 100000 Justification: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: SHARP_TH
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Elia Merzari Applicant's institution: ANL Applicant's division: NE Project Name: SHARP_TH Project title: SHARP Thermal-Hydraulics Development Associated funding: DOE-Nuclear Energy Other Systems: Science: To validate, develop and extend the thermal-hydraulics capabilities of the multi-physics nuclear reactor analysis platform package SHARP. This project comprises several activities in the SHARP thermal-hydraulics development and validation efforts in support of the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program under DOE-NE: Item 1) The commercial CFD code STAR-CCM+ and the DOE nuclear fuel performance code AMP will be coupled within the SHARP framework for multi-physics analysis of Light Water Reactors (LWRs). In the planned activities for FY12, two calculations will be performed in parallel with AMP and STAR-CCM+, comprising an LWR fuel assembly. The purpose of this item of this item is to demonstrate feasibility of such find of coupling, which is an emerging field in nuclear reactor analysis.. Item 2) Using the commercial CFD code STAR-CCM+ we plan to address the international NEA blind benchmark set forth for April 2012. The object of the benchmark is the MATiS experiment, consisting of fine-detaileda PIV measurements of the flow in a 5x5 rod bundle in the wake of a grid spacer. The objective is to validate CFD practices for this type of flow, which is practically encountered in fuel assemblies in nuclear power reactors. Given the complex geometry and highly turbulent flow, this exercise presents a highly challenging problem in terms of both the computational expense and the physics involved. Item 3) Using the commercial CFD code STAR-CCM+ we plan to study in further detail the Argonne MAX experiment. In previous years, preliminary calculations for the MAX experiment were performed. Now that funding has been secured for the finalization of the experimental activities we plan to finalize such calculations and support the experimental work. In addition this work would be instrumental in validating CFD practices for the flow configurations examined. Project description: Item 1, (STAR-CCM+, AMP coupling) STAR-CCM+ employs the finite volume formulation of the Reynolds-Averaged Navier-Stokes (RANS) method to determine the local three-dimensional velocity distribution in each sub-channel. Simultaneously, it is capable to solve the conjugate heat transfer problem to determine the temperature distribution within the fuel. This temperature data is provided to AMP to evaluate fuel performance. Initially the AMP code will be run separately and the code will be coupled off-line. Future developments will include on-line coupling using the MOAB framework. Item 2, (MATiS experiment) Several RANS simulations of the flow in the MATiS experiment rod bundle with mixing vanes (split vane and swirl type) will be performed and compared with LES simulation performed on BG/P. The optimal turbulence model and computational practices will be determined. The results will be used to participate in the international NEA benchmark. Item 3, (MAX experiment support) Using insight from previous work steady state and full transient URANS calculations will be performed. Results will be compared with available experimental data and used to determine and guide the experimental matrix. Access to the Fusion cluster would be crucial in performing all the proposed work. The STAR-CCDM+ code has already been installed on Fusion. Compilers, such as the Intel Fortran Compiler necessary to compile users subroutines are already installed. All jobs run on Fusion will be run in parallel. All calculations proposed require large computational resources due to the size of the problem. Four to ten users are expected for this project. Project URL: Requested allocation: 900000 Q1: 100000 Q2: 400000 Q3: 150000 Q4: 250000 Justification: At Argonne and elsewhere, the CFD codes STAR-CCM+ is routinely applied to simulations that use up to thousands of cores, and good parallel performance has been observed. Optimal performance is usually observed in steady state analysis is observed usually with 50,000 - 100,000 meshes per core. The typical size of the problems proposed are the following: Item 1) STAR-CCM+ - AMP coupling: 20,000,000 meshes. This would require typically 256 cores to be performed. Considering that a problem would likely converge within 48h of continouscontinuous calculation and that we would like to perform at least 20 production runs with 3,000 h of development this would translate in approximately 250,000 core-hours. Item 2) MATiS experiment simulation: 50,000,000-100,000,000 meshes. This would require typically 512-1024 cores to be performed. Considering that a problem would likely converge within 48h of continouscontinuous calculation and that we would like to perform at least 15 production runs with 3,000 h of development this would translate in approximately 400,000 core-hours. Item 3) MAX simulation: 5,000,000 meshes. This would require typically 64 cores to be performed. Long transient are likely to be necessary with 200h of consecutive calculations. Considering that we would like to perform at least 20 production transient runs, with 3,000 h of development this would translate in approximately 250,000 core-hours. The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from SOFC
by accounts＠lcrc.anl.gov 14 Sep '11

14 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Yuxin Wang Project Name: SOFC Division: XSD Project title: Imaging, Analysis and Simulation of Heterogeneous Functional Materials Associated funding: DoE Other Systems: Primarily workstations at present. Science: The DOE Energy Frontier Research Center (EFRC) on Heterogeneous Functional Materials for Energy Systems (HeteroFoaM Center), led by the University of South Carolina, is a collaboration with scientists at APS, University of Connecticut, Princeton University, University of Utah, the Georgia Institute of Technology, Rochester Institute of Technology, and the University of California-Santa Barbara. The central objective of this EFRC, funded by DOE-SC Office of Basic Energy Sciences, is to build a scientific basis for bridging the gap between making nano-structured materials and understanding how they function in a variety of energy applications (www.science.doe.gov/bes/EFRC/CENTERS/HeteroFoaM/efrc_HeteroFoaM.html) Energy devices such as fuel cells, electrolyzers, combustion and fuel processing devices consist of multiple materials interacting at multiple scales. Establishing a fundamental understanding of the functional behavior of these material systems at multiple length and time scales as the basis for providing a scientific foundation for the conceptual design, simulation and fabrication of nano-structured heterogeneous materials for energy systems is the objective of the EFRC. Central to this effort is the formulation of science-based theory and computational methods for the analysis and simulation of the functional performance of these materials based on their composition, configuration and nano-morphology. Central themes in understanding functional materials in non-equilibrium states involves understanding transport, conversion, transfer of mass, momentum, energy and charge, as well as stability and durability for long-term performance. The present proposal seeks the computational resources required for the reconstruction and analysis of materials imaged using a transmission x-ray microscope at APS beamline 32-ID-C. Structures, elemental and chemistry information obtained will be used to perform quantum chemical and mechanical, and mesoscale (phase-field and lattice Boltzmann) modeling and simulation of the non-equilibrium thermodynamics and diffusion/reaction kinetics at the atomic scale at interfaces in heterogeneous materials, stability of films as function of morphology, and predictive estimation of properties and functional behavior. The methodologies that will be applied include i) quantum-mechanical atomic level simulations to understand how materials behave at interfaces and as function of morphology, dopants, and nano-sized features, ii) mesoscale simulations to incorporate geometrical effects with atomic detail, iii) evaluation of new material properties, and iv) multiscale physics analysis of soli d oxide fuel cells (SOFCs). Project description: In the proposed work, a transmission x-ray microscope (TXM) with high resolution x-ray nanotomography will be used to obtain a detailed geometric description of SOFC electrodes at 30 nm resolution useing a tunable synchrotron light source at APS beamline 32-ID-C. The XCT technique is a non-destructive method used to image and reconstruct actual SOFC microstructures. X-rays penetrate the sample mounted on a rotation stage, then passes through a scintillation screen and captured by a camera. The sample is rotated through 180 degrees with sample projections produced at specified angle intervals and exposure times. Images are captured over a field of view of up to 100 um by using stitching methods and reconstructed into a 3-D volume of the pore structure. To achieve elemental sensitivity for Ni at the Ni-YSZ interface, the microscopy experiments were carried at 8.317 keV and 8.357 keV, corresponding to 16 eV below and 24 eV above the Ni K-edge (8.333 keV), r espectively. Once the TXM data has been obtained, slice-by-slice reconstruction will be carried out to convert the image data to digitized 3D reconstructed volumes. This approach has been successfully validated. The digitized 3D geometry will be used as model input. The nanotomography data obtained require further processing before the microstructure of the samples can be fully characterized. Chromatic focusing of the TXM zoneplate lens during the measurements can cause non-negligible variation in the magnification of the sample over the energy range investigated. The images were scaled to eliminate this variation. The transmission images from two different energy levels were processed to isolate the Ni phases to produce two separate nanotomography data sets, one representing Ni and another phase. Each of these data sets was aligned on a central rotation axis, and reconstructed using the iterative algebraic reconstruction technique (i-ART) algorithm. The output from the reconstruction is two image stacks corresponding to each phase of interest. These stacks are processed separately with segmentation steps performed using the ImageJ software. After performing fine alignment to account for minor variations in stage positioning, the p rocessed stacks are merged to produce one final volumetric data set, with discrete identification of each phase. This volumetric data is used as an input to previously developed microstructural characterization codes. Image processing is a compute intensive task, but necessary to obtain useful structures and elemental/chemical information for quantum chemical simulations, atomistic simulations, and mesoscale phase-field and lattice Boltzmann simulations. Quantum chemical simulations and Quantum mechanical calculations It is the objective of the quantum chemical calculations for this project to rationally design new nano-structured heterogeneous functional materials from the "atom-up" and to identify the scientific relationship between structure and property/function. Quantum chemical simulations will be based on density functional theory (DFT) and transition state theory which permits connecting atomistic information with meso- and macroscopic information and therefore ultimately the simulation of materials at experimental time and length scales. For the anode electrode of the SOFC, key properties of interest are high electronic conductivity, oxygen ion mobility, tolerance to sulfur impurities in the fuel stream, and tolerance to carbon poisoning. For the cathode electrode, key properties are high electronic conductivity, oxygen ion mobility, and oxygen dissociation rate. We aim at identifying a minimal set of independent descriptors for these material properties that allows rational m aterial selection, particularly dopant selection. Unfortunately, no one quantum chemical method can calculate accurately all properties of interest. The workhorse method for condensed matter electronic structure calculations at present is DFT within the generalized gradient approximation (GGA) to electron exchange and correlation. And while the SOFC electrolyte, YSZ, is well described by DFT-GGA, many materials of interest for the anode and cathode electrode of the SOFC are reduced oxides with transition metal or lanthanide ions with open shell 3d- and 4f- electrons that are described incorrectly by DFT-GGA because their approximate exchange energies do not exactly cancel the spurious Coulomb repulsion of each open-shell d-electron with itself, leading to the so-called "self-interaction error (SIE)". The consequences of the inexact cancellation are not just quantitative but qualitative failure, e.g., with semiconductors sometimes predicted to be metals (e.g., NiO and FeO). This class of materials is referred to as strong ly correlated, and cannot be described properly in a static mean field theory, such as DFT. Although several first principles quantum mechanics methods of treating such materials exist, we will employ electronic structure methods that provide the correct physics at minimal extra cost beyond a conventional DFT calculation. These are the periodic electrostatic embedded cluster (PEEC) method with hybrid and double-hybrid density functionals and the DFT+U method, originally proposed by Anisimov, Lichtenstein and coworkers,[1-3] in which the spurious intra-atomic self-Coulomb repulsion of the open-shell d and/or f electrons is cancelled by using exact intra-atomic Hartree-Fock (HF) exchange, while the rest of the electrons are described within DFT. Materials of interest for the anode side of the SOFC are doped perovskite and double perovskite structures such as SrTiO3 and Sr2Fe1.5Mo0.5O6-δ. Double perovskite structures based on Mo can easily be doped with various transition metals, permitting a significant variation of the material properties. Materials of interest for the cathode side are gadolinium-doped ceria (Gd-doped CeO2 or GDC), lanthanum strontium cobaltate (La0.6Sr0.4CoO3 LSC, the Fe-doped analog (LSCF), Sr2Fe1.5Mo0.5O6-δ, and lanthanum strontium manganate (LSM). We will employ DFT+U based calculations to characterize the bulk and surface electronic structure and kinetic processes of the SOFC cathode materials LSCF and LSM. DFT+U requires as input a parameter that is the difference between the averaged Coulomb (U) and exchange (J) interactions for the electrons of a specific angular momentum localized on each atomic site. We use the method of Mosey, Liao, and Carter[4] for calculating U and J ab initio from unrestricted Hartree-Fock (UHF) calculations on electrostatically embedded clusters, so as to render the DFT+U method nonempirical. We are interested in improving oxygen ion transport in mixed electron-ion conductors such as LSCF and LSM by doping or alloying these parent materials. To predict the kinetics of oxygen diffusion through the SOFC cathode material, we will employ the nudged elastic band (NEB)[5] method to locate the minimum energy path for diffusion. This technique determines the transition state from which we can calculate the activation energy and, by calculating the vibrational frequencies associated with the minimum and transition state, the pre-exponential factor for the diffusion constant.[6-7] DFT with the GGA and LDA functionals also fail to accurately describe the bond energy of dioxygen. This failure occurs not only because of the SIE mentioned earlier but also the open-shell triplet ground state of O2 is not a pure spin eigenfunction of the single determinant wavefunction employed in DFT. To overcome this failure, we will employ embedded configuration interaction (ECI)[8] calculations. Here the system is broken into a small region of interest treated with a highly accurate quantum mechanical method (e.g., multi-reference single and double excitation CI, MRSDCI) and a background region described with a lower-level method (DFT+U or electrostatic embedding) that defines an embedding potential for the region of interest. In addition to providing an accurate description of oxygen bond dissociation energetics, this method allows for the explicit inclusion of an electron at the LSCF and LSM surface. In this way, we can model the electrochemical reduction processes occurring at the SOFC cathode. Using these methods, we aim to predict the kinetics of the different cathode processes which influence the overall performance of the SOFC. From our calculations, we will seek to understand how material composition affects the different kinetic parameters as well as determining which processes may serve as key descriptors of the overall cathode kinetics. Mesoscale Phase-field Simulations In order to address the issue of microstructural stability in SOFCs over extended operation times, a coarse-grained continuum approach is required. The so-called "diffuse-interface" or "phase-field" (PF) modeling framework, in which Prof. Mikko Haataja has extensive experience[12,13,14], has become very popular over the past several years as a tool to simulate evolving microstructures in multicomponent, multiphase materials. In this approach, several continuum fields ("order parameters") are first introduced to account for both the local phase (solid vs. pore) and composition/structure (Ni vs. YSZ). Then, a thermodynamic free energy, which accounts for the thermodynamics of bulk phases and relevant interfacial energies, is constructed. Finally, the dynamics of the order parameters are obtained from the free energy via variational principles, leading to coupled non-linear partial differential equations defined in the whole computational domain. One of the strengths of the PF method is that it conveniently handles complex topological changes occurring during the microstructural evolution process, while the numerical challenge is to integrate stiff PDEs over extended simulation times in a 3D computational domain. Lattice Boltzmann calculations Over the past decade, the lattice Boltzmann method (LBM) has been applied to solve an astonishingly large variety of problems in various disciplines, particularly at the mesoscale. LBM techniques are CPU and memory intensive. This often restricts the lattice resolution that can be used on standard cluster computers. Now that massively parallel computers such as the FUSION system are available, these CPU and memory issues can be addressed so problems that were previously intractable using the LBM can be brought within reach. The LBM scheme has been shown to scale well on large computing systems. Excellent scale-up was observed up to 1024 processors by Oliker et al. using a 512^3 lattice to study flow turbulence[15]. For a turbulence application, a sustained performance of 2 TFlops was achieved on 256 NEC vector CPUs. This indicates that the LBM is a very promising candidate for large scale computing platforms like FUSION. Mathematical modeling of transport phenomenon in a fuel cell is very useful for analyzing and optimizing fuel cells. For example, team member Prof. Wilson K. S. Chiu has used three-dimensional (3D) LBM models to investigate multi-component mass transport with electrochemistry and reformation in porous SOFCs[16-18]. The imaging technology required to reconstruct the 3D SOFC electrode geometry is available[19], and 3D LBM simulations would significantly increase the accuracy of the SOFC mass transport model. The goal of our research is to model the diffusive flow of three gaseous species (H2, H2O, and N2) through the porous anode structure of a SOFC. Our model uses a 3D representation of the porous anode, obtained from x-ray nanotomography at APS beamline 32-ID-C, with resolution as low as 30 nm[19]. Such analysis is often very expensive on scalar and smaller cluster systems and often restricts the resolution that can be used to model the problem of interest. However, now that massively parallel computers such as FUSION are available, these memory issues can be addressed and problems that were intractable using the LBM only a few years ago are within reach. References 1. V. Anisimov, J. Zaanen, O. K. Andersen, "Band theory and Mott insulators: Hubbard U instead of Stoner I", Physical Review B, vol. 44, pp. 943-954 (1991). 2. V. Anisimov, F. Aryasetiawan, A. I. Lichtenstein, "First-principles calculations of the electronic structure and spectra of strongly correlated systems: the LDA+U method", Journal of Physics: Condensed Matter, vol. 9, pp. 767-808 (1997). 3. S. L. Dudarev, A. I. Lichtenstein, M. R. Castell, G. A. D. Briggs, A. P. Sutton, "Surface states on NiO (100) and the origin of the contrast reversal in atomically resolved scanning tunneling microscope images", Physical Review B, vol. 56, pp. 4900-4908 (1997). 4. N. J. Mosey, P. Liao, E. A. Carter, "Rotationally invariant ab initio evaluation of Coulomb and exchange parameters for DFT+U calculations", Journal of Chemical Physics, vol. 129, pp. 014103 (2008). 5. H. Jo'nsson, G. Mills, K. Jacobsen, "Nudged elastic band method for finding minimum energy paths of transitions" in Classical and Quantum Dynamics in Condensed Phase Simulations, pp. 385-404, World Scientific, 1998. 6. K. A. Marino and E. A. Carter, "The effect of platinum on diffusion kinetics is beta-NiAl: Implications for thermal barrier coating lifetimes", ChemPhysChem, vol. 10, pp. 226, (2009). 7. K. A. Marino and E. A. Carter, "First-principles characterization of Ni diffusion kinetics in beta-NiAl", Physical Review B, vol. 78, pp. 184105 (2008). 8. P. Huang and E. A. Carter, "Self-consistent embedding theory for locally correlated configuration interaction wave functions in condensed matter", Journal of Chemical Physics, vol. 125, pp. 084102 (2006). 9. R. M. Shroll, and T. P. Straatsma, "Molecular Dynamics Simulations of the Goethite-Water Interface", Molecular Simulation, 29, 1-11 (2003) 10. R. M. Shroll, and T. P. Straatsma, "Molecular Basis for Microbial Adhesion to Geochemical Surfaces: Computer Simulation of Pseudomonas aeruginosa Adhesion to Goethite", Biophysical Journal, 84, 1765-1772 (2003). 11. T. P. Straatsma and J. A. McCammon, "Multiconfiguration thermodynamic integration", J. Chem. Phys., 95, 1175-1188 (1991). 12. S. Sreekala and M. Haataja, "Recrystallization kinetics: A coupled coarse-grained dislocation density and phase-field approach", Phys. Rev. B 76, 094109 (2007). 13. M. Haataja and F. Le'onard, "Spinodal decomposition in the presence of mobile dislocations", Phys. Rev. B 69, 081201 (R) (2004). 14. M. Haataja, J. Mueller, A. D. Rutenberg, and M. Grant, "Dislocations and morphological instabilities: Continuum modeling of misfitting heteroepitaxial films", Phys. Rev. B 65, 165414 (2002). 15. L. Oliker, A. Canning, J. Carter, C. Iancu, M. Lijewski, S. Kamil, J. Shalf, H. Shan, E. Strohmaier, S. Ethier, T. Goodale (2007) Scientific Application Performance on Candidate Petascale Platforms, International Parallel and Distributed Processing Symposium (IPDPS), March 24 – 30, 2007, Long Beach, CA. 16. A. S. Joshi, K. N. Grew, J. R. Izzo, Jr., A. A. Peracchio, and W. K. S. Chiu, "Lattice Boltzmann Modeling of Three-Dimensional, Multicomponent Mass Diffusion in a Solid Oxide Fuel Cell Anode," ASME Journal of Fuel Cell Science and Technology, vol. 7, pp. 011006-1-011006-8 (2010). 17. A. S. Joshi, K. N. Grew, A. A. Peracchio and W. K. S. Chiu, "Lattice Boltzmann Modeling of 2D Gas Transport in a Solid Oxide Fuel Cell Anode," Journal of Power Sources, vol. 164, pp. 631-638 (2007). 18. K. N. Grew, A. S. Joshi, A. A. Peracchio, and W. K. S. Chiu, "Pore-Scale Investigation of Mass Transport and Electrochemistry in a Solid Oxide Fuel Cell Anode," Journal of Power Sources, vol. 195, pp. 2331–2345 (2010). 19. K. N. Grew, Y. S. Chu, J. Yi, A. A. Peracchio, J. R. Izzo, Jr., F. De Carlo, Y. Hwu and W. K. S. Chiu, "Nondestructive Nanoscale 3D Elemental Mapping and Analysis of a Solid Oxide Fuel Cell Anode," Journal of the Electrochemical Society, vol. 157(5), (2010). 20. G. Kresse and J. Furthmueller, "Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set", Computational Materials Science, vol. 6, pp. 15-50, 1996. 21. G. Kresse and J. Furthmueller, "Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set", Physical Review B, vol. 54, pp. 11169-11186 (1996). 22. Peters, B., Heyden, A., Bell, A. T., Chakraborty, A., A growing string method for determining transition states: Comparison to the nudged elastic band and string methods. J. Chem. Phys. 120, 7877-7886 (2004). 23. Heyden, A., Bell, A. T., Keil, F. J., Efficient methods for finding transition states in chemical reactions: Comparison of modified dimer method and partitioned rational function optimization method. J. Chem. Phys. 123, 224101-14 (2005). Project URL: http://www.science.doe.gov/bes/EFRC/CENTERS/HeteroFoaM/efrc_HeteroFoaM.html Current FY Hours Used: undetermined amount New FY Requested allocation: 2650000 Q1: 600000 Q2: 650000 Q3: 700000 Q4: 700000 Justification: Computational Resource Needs – September 2011 Image processing All image processing will be performed using Matlab. We will reconstruct an average of 20 samples per week requiring an average of 48 node hours per calculation (50,000 node hours per year). We will also be performing additional image processing of each sample requiring approximately 30 node hours per calculation (30,000 node hours per year). All calculations will require up to 100 GB of memory. Quantum chemical simulations & Quantum mechanical calculations Our DFT calculations will be performed using the Vienna Ab Initio Simulation Package (VASP) which solves the Kohn-Sham equations subject to periodic boundary conditions using a planewave basis set. We will perform an average of 40 single-point energy calculations each week requiring an average of 48 node hours per calculation (99,840 node hours per year). We will also be performing an average of 3 nudged elastic band calculations each week requiring approximately 1,200 node hours per calculation (187,200 node hours per year). All PEECM calculations will be performed with the Turbomole code (based on Gaussian-type basis sets) using hybrid and double-hybrid density functionals. Transition states required for the calculation of reaction rates will be located with our own highly efficient growing string and improved dimer algorithms. Recent tests on EMSL Chinook confirmed that both VASP and Turbomole scale well up to hundreds of nodes. Overall, we expect that computations required for quantum chemical calculations on the anode side of the SOFC will require approximately 750,000 node-hours per year and 500 GB of memory. This estimate is based on submitting 150 geometry optimizations and 20 transition state searches per year for each the perovskite and double perovskite structures (optimization of a perovskite structure requires on FUSION about 5000 node hours; optimization for a double perovskite structure requires about 12,500 node hours; transition state searches require about 10 times more time than a structure optimization). Phase-field Simulations Our continuum PF simulations will be performed with a parallel MPI code written in-house. While we have not benchmarked this particular code, our experience with similar codes in the past suggests that it should scale well up to 32 or 64 CPUs on the FUSION system. Furthermore, we estimate that a simulation with 1283 (2563) grid points will require 240 (1920) node-hours wall clock time and 100 GB (800GB) of memory. On an annual basis, we expect that our simulations of SOFC anode microstructural evolution phenomena require on the order of 200,000 node hours and 1 TB of memory on the FUSION system, corresponding to 100 high-resolution simulations. Lattice Boltzmann Calculations Our LBM simulation of 3D mass transport and electrochemistry in solid oxide fuel cells scales well up to 64 CPU on a local SGI Altix 850 system. For the FUSION system, we estimate that a single case using low resolution 813 lattice will require 350 node-hours wall-clock time and 100 GB memory on the FUSION system. A high resolution 6743 lattice will enable us to simulate realistic physical domain sizes and lead to better accuracy of the model predictions, but each case will require 22,400 node-hours wall-clock time and 1 TB memory on the FUSION system. Our annual estimated EFRC project needs on the FUSION system are 300,000 node hours: two low resolution lattice cases per week (36,400 node-hours wall-clock time per year), and one high resolution lattice case per month (268,800 node-hours wall-clock time per year). Summary of Computing Resource Needs (September 2011 to March 2012) Software Requirements Node Hours Storage GB Image Processing Matlab 30,000 100 Quantum chemical simulations Turbomole 140,000 500 Quantum mechanical calculations VASP 287,000 500 Phase-field simulations Intel C++ Compiler 100,000 1500 Lattice Boltzmann calculations Intel Fortran Compiler 45,000 1500 Total 602,000 3000 Thank You, The LCRC Accounts System

1 0