Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Yuxin Wang Project Name: SOFC Division: XSD Project title: Imaging, Analysis and Simulation of Heterogeneous Functional Materials Associated funding: DoE Other Systems: Primarily workstations at present. Science: The DOE Energy Frontier Research Center (EFRC) on Heterogeneous Functional Materials for Energy Systems (HeteroFoaM Center), led by the University of South Carolina, is a collaboration with scientists at APS, University of Connecticut, Princeton University, University of Utah, the Georgia Institute of Technology, Rochester Institute of Technology, and the University of California-Santa Barbara. The central objective of this EFRC, funded by DOE-SC Office of Basic Energy Sciences, is to build a scientific basis for bridging the gap between making nano-structured materials and understanding how they function in a variety of energy applications (www.science.doe.gov/bes/EFRC/CENTERS/HeteroFoaM/efrc_HeteroFoaM.html). Energy devices such as fuel cells, electrolyzers, combustion and fuel processing devices consist of multiple materials interacting at multiple scales. Establishing a fundamental understanding of the functional behavior of these material systems at multiple length and time scales as the basis for providing a scientific foundation for the conceptual design, simulation and fabrication of nano-structured heterogeneous materials for energy systems is the objective of the EFRC. Central to this effort is the formulation of science-based theory and computational methods for the analysis and simulation of the functional performance of these materials based on their composition, configuration and nano-morphology. Central themes in understanding functional materials in non-equilibrium states involves understanding transport, conversion, transfer of mass, momentum, energy and charge, as well as stability and durability for long-term performance. The present proposal seeks the computational resources required for the reconstruction and analysis of materials imaged using a transmission x-ray microscope at APS beamline 32-ID-C. Structures, elemental and chemistry information obtained will be used to perform quantum chemical and mechanical, and mesoscale (phase-field and lattice Boltzmann) modeling and simulation of the non-equilibrium thermodynamics and diffusion/reaction kinetics at the atomic scale at interfaces in heterogeneous materials, stability of films as function of morphology, and predictive estimation of properties and functional behavior. The methodologies that will be applied include i) quantum-mechanical atomic level simulations to understand how materials behave at interfaces and as function of morphology, dopants, and nano-sized features, ii) mesoscale simulations to incorporate geometrical effects with atomic detail, iii) evaluation of new material properties, and iv) multiscale physics analysis of soli d oxide fuel cells (SOFCs). Project description: In the proposed work, a transmission x-ray microscope (TXM) with high resolution x-ray nanotomography will be used to obtain a detailed geometric description of SOFC electrodes at 30 nm resolution useing a tunable synchrotron light source at APS beamline 32-ID-C. The XCT technique is a non-destructive method used to image and reconstruct actual SOFC microstructures. X-rays penetrate the sample mounted on a rotation stage, then passes through a scintillation screen and captured by a camera. The sample is rotated through 180 degrees with sample projections produced at specified angle intervals and exposure times. Images are captured over a field of view of up to 100 um by using stitching methods and reconstructed into a 3-D volume of the pore structure. To achieve elemental sensitivity for Ni at the Ni-YSZ interface, the microscopy experiments were carried at 8.317 keV and 8.357 keV, corresponding to 16 eV below and 24 eV above the Ni K-edge (8.333 keV), r espectively. Once the TXM data has been obtained, slice-by-slice reconstruction will be carried out to convert the image data to digitized 3D reconstructed volumes. This approach has been successfully validated. The digitized 3D geometry will be used as model input. The nanotomography data obtained require further processing before the microstructure of the samples can be fully characterized. Chromatic focusing of the TXM zoneplate lens during the measurements can cause non-negligible variation in the magnification of the sample over the energy range investigated. The images were scaled to eliminate this variation. The transmission images from two different energy levels were processed to isolate the Ni phases to produce two separate nanotomography data sets, one representing Ni and another phase. Each of these data sets was aligned on a central rotation axis, and reconstructed using the iterative algebraic reconstruction technique (i-ART) algorithm. The output from the reconstruction is two image stacks corresponding to each phase of interest. These stacks are processed separately with segmentation steps performed using the ImageJ software. After performing fine alignment to account for minor variations in stage positioning, the p rocessed stacks are merged to produce one final volumetric data set, with discrete identification of each phase. This volumetric data is used as an input to previously developed microstructural characterization codes. Image processing is a compute intensive task, but necessary to obtain useful structures and elemental/chemical information for quantum chemical simulations, atomistic simulations, and mesoscale phase-field and lattice Boltzmann simulations. Quantum chemical simulations and Quantum mechanical calculations It is the objective of the quantum chemical calculations for this project to rationally design new nano-structured heterogeneous functional materials from the "atom-up" and to identify the scientific relationship between structure and property/function. Quantum chemical simulations will be based on density functional theory (DFT) and transition state theory which permits connecting atomistic information with meso- and macroscopic information and therefore ultimately the simulation of materials at experimental time and length scales. For the anode electrode of the SOFC, key properties of interest are high electronic conductivity, oxygen ion mobility, tolerance to sulfur impurities in the fuel stream, and tolerance to carbon poisoning. For the cathode electrode, key properties are high electronic conductivity, oxygen ion mobility, and oxygen dissociation rate. We aim at identifying a minimal set of independent descriptors for these material properties that allows rational m aterial selection, particularly dopant selection. Unfortunately, no one quantum chemical method can calculate accurately all properties of interest. The workhorse method for condensed matter electronic structure calculations at present is DFT within the generalized gradient approximation (GGA) to electron exchange and correlation. And while the SOFC electrolyte, YSZ, is well described by DFT-GGA, many materials of interest for the anode and cathode electrode of the SOFC are reduced oxides with transition metal or lanthanide ions with open shell 3d- and 4f- electrons that are described incorrectly by DFT-GGA because their approximate exchange energies do not exactly cancel the spurious Coulomb repulsion of each open-shell d-electron with itself, leading to the so-called "self-interaction error (SIE)". The consequences of the inexact cancellation are not just quantitative but qualitative failure, e.g., with semiconductors sometimes predicted to be metals (e.g., NiO and FeO). This class of materials is referred to as strong ly correlated, and cannot be described properly in a static mean field theory, such as DFT. Although several first principles quantum mechanics methods of treating such materials exist, we will employ electronic structure methods that provide the correct physics at minimal extra cost beyond a conventional DFT calculation. These are the periodic electrostatic embedded cluster (PEEC) method with hybrid and double-hybrid density functionals and the DFT+U method, originally proposed by Anisimov, Lichtenstein and coworkers,[1-3] in which the spurious intra-atomic self-Coulomb repulsion of the open-shell d and/or f electrons is cancelled by using exact intra-atomic Hartree-Fock (HF) exchange, while the rest of the electrons are described within DFT. Materials of interest for the anode side of the SOFC are doped perovskite and double perovskite structures such as SrTiO3 and Sr2Fe1.5Mo0.5O6-δ. Double perovskite structures based on Mo can easily be doped with various transition metals, permitting a significant variation of the material properties. Materials of interest for the cathode side are gadolinium-doped ceria (Gd-doped CeO2 or GDC), lanthanum strontium cobaltate (La0.6Sr0.4CoO3 LSC, the Fe-doped analog (LSCF), Sr2Fe1.5Mo0.5O6-δ, and lanthanum strontium manganate (LSM). We will employ DFT+U based calculations to characterize the bulk and surface electronic structure and kinetic processes of the SOFC cathode materials LSCF and LSM. DFT+U requires as input a parameter that is the difference between the averaged Coulomb (U) and exchange (J) interactions for the electrons of a specific angular momentum localized on each atomic site. We use the method of Mosey, Liao, and Carter[4] for calculating U and J ab initio from unrestricted Hartree-Fock (UHF) calculations on electrostatically embedded clusters, so as to render the DFT+U method nonempirical. We are interested in improving oxygen ion transport in mixed electron-ion conductors such as LSCF and LSM by doping or alloying these parent materials. To predict the kinetics of oxygen diffusion through the SOFC cathode material, we will employ the nudged elastic band (NEB)[5] method to locate the minimum energy path for diffusion. This technique determines the transition state from which we can calculate the activation energy and, by calculating the vibrational frequencies associated with the minimum and transition state, the pre-exponential factor for the diffusion constant.[6-7] DFT with the GGA and LDA functionals also fail to accurately describe the bond energy of dioxygen. This failure occurs not only because of the SIE mentioned earlier but also the open-shell triplet ground state of O2 is not a pure spin eigenfunction of the single determinant wavefunction employed in DFT. To overcome this failure, we will employ embedded configuration interaction (ECI)[8] calculations. Here the system is broken into a small region of interest treated with a highly accurate quantum mechanical method (e.g., multi-reference single and double excitation CI, MRSDCI) and a background region described with a lower-level method (DFT+U or electrostatic embedding) that defines an embedding potential for the region of interest. In addition to providing an accurate description of oxygen bond dissociation energetics, this method allows for the explicit inclusion of an electron at the LSCF and LSM surface. In this way, we can model the electrochemical reduction processes occurring at the SOFC cathode. Using these methods, we aim to predict the kinetics of the different cathode processes which influence the overall performance of the SOFC. From our calculations, we will seek to understand how material composition affects the different kinetic parameters as well as determining which processes may serve as key descriptors of the overall cathode kinetics. Mesoscale Phase-field Simulations In order to address the issue of microstructural stability in SOFCs over extended operation times, a coarse-grained continuum approach is required. The so-called "diffuse-interface" or "phase-field" (PF) modeling framework, in which Prof. Mikko Haataja has extensive experience[12,13,14], has become very popular over the past several years as a tool to simulate evolving microstructures in multicomponent, multiphase materials. In this approach, several continuum fields ("order parameters") are first introduced to account for both the local phase (solid vs. pore) and composition/structure (Ni vs. YSZ). Then, a thermodynamic free energy, which accounts for the thermodynamics of bulk phases and relevant interfacial energies, is constructed. Finally, the dynamics of the order parameters are obtained from the free energy via variational principles, leading to coupled non-linear partial differential equations defined in the whole computational domain. One of the strengths of the PF method is that it conveniently handles complex topological changes occurring during the microstructural evolution process, while the numerical challenge is to integrate stiff PDEs over extended simulation times in a 3D computational domain. Lattice Boltzmann calculations Over the past decade, the lattice Boltzmann method (LBM) has been applied to solve an astonishingly large variety of problems in various disciplines, particularly at the mesoscale. LBM techniques are CPU and memory intensive. This often restricts the lattice resolution that can be used on standard cluster computers. Now that massively parallel computers such as the FUSION system are available, these CPU and memory issues can be addressed so problems that were previously intractable using the LBM can be brought within reach. The LBM scheme has been shown to scale well on large computing systems. Excellent scale-up was observed up to 1024 processors by Oliker et al. using a 512^3 lattice to study flow turbulence[15]. For a turbulence application, a sustained performance of 2 TFlops was achieved on 256 NEC vector CPUs. This indicates that the LBM is a very promising candidate for large scale computing platforms like FUSION. Mathematical modeling of transport phenomenon in a fuel cell is very useful for analyzing and optimizing fuel cells. For example, team member Prof. Wilson K. S. Chiu has used three-dimensional (3D) LBM models to investigate multi-component mass transport with electrochemistry and reformation in porous SOFCs[16-18]. The imaging technology required to reconstruct the 3D SOFC electrode geometry is available[19], and 3D LBM simulations would significantly increase the accuracy of the SOFC mass transport model. The goal of our research is to model the diffusive flow of three gaseous species (H2, H2O, and N2) through the porous anode structure of a SOFC. Our model uses a 3D representation of the porous anode, obtained from x-ray nanotomography at APS beamline 32-ID-C, with resolution as low as 30 nm[19]. Such analysis is often very expensive on scalar and smaller cluster systems and often restricts the resolution that can be used to model the problem of interest. However, now that massively parallel computers such as FUSION are available, these memory issues can be addressed and problems that were intractable using the LBM only a few years ago are within reach. References 1. V. Anisimov, J. Zaanen, O. K. Andersen, "Band theory and Mott insulators: Hubbard U instead of Stoner I", Physical Review B, vol. 44, pp. 943-954 (1991). 2. V. Anisimov, F. Aryasetiawan, A. I. Lichtenstein, "First-principles calculations of the electronic structure and spectra of strongly correlated systems: the LDA+U method", Journal of Physics: Condensed Matter, vol. 9, pp. 767-808 (1997). 3. S. L. Dudarev, A. I. Lichtenstein, M. R. Castell, G. A. D. Briggs, A. P. Sutton, "Surface states on NiO (100) and the origin of the contrast reversal in atomically resolved scanning tunneling microscope images", Physical Review B, vol. 56, pp. 4900-4908 (1997). 4. N. J. Mosey, P. Liao, E. A. Carter, "Rotationally invariant ab initio evaluation of Coulomb and exchange parameters for DFT+U calculations", Journal of Chemical Physics, vol. 129, pp. 014103 (2008). 5. H. Jo'nsson, G. Mills, K. Jacobsen, "Nudged elastic band method for finding minimum energy paths of transitions" in Classical and Quantum Dynamics in Condensed Phase Simulations, pp. 385-404, World Scientific, 1998. 6. K. A. Marino and E. A. Carter, "The effect of platinum on diffusion kinetics is beta-NiAl: Implications for thermal barrier coating lifetimes", ChemPhysChem, vol. 10, pp. 226, (2009). 7. K. A. Marino and E. A. Carter, "First-principles characterization of Ni diffusion kinetics in beta-NiAl", Physical Review B, vol. 78, pp. 184105 (2008). 8. P. Huang and E. A. Carter, "Self-consistent embedding theory for locally correlated configuration interaction wave functions in condensed matter", Journal of Chemical Physics, vol. 125, pp. 084102 (2006). 9. R. M. Shroll, and T. P. Straatsma, "Molecular Dynamics Simulations of the Goethite-Water Interface", Molecular Simulation, 29, 1-11 (2003) 10. R. M. Shroll, and T. P. Straatsma, "Molecular Basis for Microbial Adhesion to Geochemical Surfaces: Computer Simulation of Pseudomonas aeruginosa Adhesion to Goethite", Biophysical Journal, 84, 1765-1772 (2003). 11. T. P. Straatsma and J. A. McCammon, "Multiconfiguration thermodynamic integration", J. Chem. Phys., 95, 1175-1188 (1991). 12. S. Sreekala and M. Haataja, "Recrystallization kinetics: A coupled coarse-grained dislocation density and phase-field approach", Phys. Rev. B 76, 094109 (2007). 13. M. Haataja and F. Le'onard, "Spinodal decomposition in the presence of mobile dislocations", Phys. Rev. B 69, 081201 (R) (2004). 14. M. Haataja, J. Mueller, A. D. Rutenberg, and M. Grant, "Dislocations and morphological instabilities: Continuum modeling of misfitting heteroepitaxial films", Phys. Rev. B 65, 165414 (2002). 15. L. Oliker, A. Canning, J. Carter, C. Iancu, M. Lijewski, S. Kamil, J. Shalf, H. Shan, E. Strohmaier, S. Ethier, T. Goodale (2007) Scientific Application Performance on Candidate Petascale Platforms, International Parallel and Distributed Processing Symposium (IPDPS), March 24 – 30, 2007, Long Beach, CA. 16. A. S. Joshi, K. N. Grew, J. R. Izzo, Jr., A. A. Peracchio, and W. K. S. Chiu, "Lattice Boltzmann Modeling of Three-Dimensional, Multicomponent Mass Diffusion in a Solid Oxide Fuel Cell Anode," ASME Journal of Fuel Cell Science and Technology, vol. 7, pp. 011006-1-011006-8 (2010). 17. A. S. Joshi, K. N. Grew, A. A. Peracchio and W. K. S. Chiu, "Lattice Boltzmann Modeling of 2D Gas Transport in a Solid Oxide Fuel Cell Anode," Journal of Power Sources, vol. 164, pp. 631-638 (2007). 18. K. N. Grew, A. S. Joshi, A. A. Peracchio, and W. K. S. Chiu, "Pore-Scale Investigation of Mass Transport and Electrochemistry in a Solid Oxide Fuel Cell Anode," Journal of Power Sources, vol. 195, pp. 2331–2345 (2010). 19. K. N. Grew, Y. S. Chu, J. Yi, A. A. Peracchio, J. R. Izzo, Jr., F. De Carlo, Y. Hwu and W. K. S. Chiu, "Nondestructive Nanoscale 3D Elemental Mapping and Analysis of a Solid Oxide Fuel Cell Anode," Journal of the Electrochemical Society, vol. 157(5), (2010). 20. G. Kresse and J. Furthmueller, "Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set", Computational Materials Science, vol. 6, pp. 15-50, 1996. 21. G. Kresse and J. Furthmueller, "Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set", Physical Review B, vol. 54, pp. 11169-11186 (1996). 22. Peters, B., Heyden, A., Bell, A. T., Chakraborty, A., A growing string method for determining transition states: Comparison to the nudged elastic band and string methods. J. Chem. Phys. 120, 7877-7886 (2004). 23. Heyden, A., Bell, A. T., Keil, F. J., Efficient methods for finding transition states in chemical reactions: Comparison of modified dimer method and partitioned rational function optimization method. J. Chem. Phys. 123, 224101-14 (2005). Project URL: http://www.science.doe.gov/bes/EFRC/CENTERS/HeteroFoaM/efrc_HeteroFoaM.html Current FY Hours Used: undetermined amount New FY Requested allocation: 2650000 Q1: 600000 Q2: 650000 Q3: 700000 Q4: 700000 Justification: Computational Resource Needs – September 2011 Image processing All image processing will be performed using Matlab. We will reconstruct an average of 20 samples per week requiring an average of 48 node hours per calculation (50,000 node hours per year). We will also be performing additional image processing of each sample requiring approximately 30 node hours per calculation (30,000 node hours per year). All calculations will require up to 100 GB of memory. Quantum chemical simulations & Quantum mechanical calculations Our DFT calculations will be performed using the Vienna Ab Initio Simulation Package (VASP) which solves the Kohn-Sham equations subject to periodic boundary conditions using a planewave basis set. We will perform an average of 40 single-point energy calculations each week requiring an average of 48 node hours per calculation (99,840 node hours per year). We will also be performing an average of 3 nudged elastic band calculations each week requiring approximately 1,200 node hours per calculation (187,200 node hours per year). All PEECM calculations will be performed with the Turbomole code (based on Gaussian-type basis sets) using hybrid and double-hybrid density functionals. Transition states required for the calculation of reaction rates will be located with our own highly efficient growing string and improved dimer algorithms. Recent tests on EMSL Chinook confirmed that both VASP and Turbomole scale well up to hundreds of nodes. Overall, we expect that computations required for quantum chemical calculations on the anode side of the SOFC will require approximately 750,000 node-hours per year and 500 GB of memory. This estimate is based on submitting 150 geometry optimizations and 20 transition state searches per year for each the perovskite and double perovskite structures (optimization of a perovskite structure requires on FUSION about 5000 node hours; optimization for a double perovskite structure requires about 12,500 node hours; transition state searches require about 10 times more time than a structure optimization). Phase-field Simulations Our continuum PF simulations will be performed with a parallel MPI code written in-house. While we have not benchmarked this particular code, our experience with similar codes in the past suggests that it should scale well up to 32 or 64 CPUs on the FUSION system. Furthermore, we estimate that a simulation with 1283 (2563) grid points will require 240 (1920) node-hours wall clock time and 100 GB (800GB) of memory. On an annual basis, we expect that our simulations of SOFC anode microstructural evolution phenomena require on the order of 200,000 node hours and 1 TB of memory on the FUSION system, corresponding to 100 high-resolution simulations. Lattice Boltzmann Calculations Our LBM simulation of 3D mass transport and electrochemistry in solid oxide fuel cells scales well up to 64 CPU on a local SGI Altix 850 system. For the FUSION system, we estimate that a single case using low resolution 813 lattice will require 350 node-hours wall-clock time and 100 GB memory on the FUSION system. A high resolution 6743 lattice will enable us to simulate realistic physical domain sizes and lead to better accuracy of the model predictions, but each case will require 22,400 node-hours wall-clock time and 1 TB memory on the FUSION system. Our annual estimated EFRC project needs on the FUSION system are 300,000 node hours: two low resolution lattice cases per week (36,400 node-hours wall-clock time per year), and one high resolution lattice case per month (268,800 node-hours wall-clock time per year). Summary of Computing Resource Needs (September 2011 to March 2012) Software Requirements Node Hours Storage GB Image Processing Matlab 30,000 100 Quantum chemical simulations Turbomole 140,000 500 Quantum mechanical calculations VASP 287,000 500 Phase-field simulations Intel C++ Compiler 100,000 1500 Lattice Boltzmann calculations Intel Fortran Compiler 45,000 1500 Total 602,000 3000 Thank You, The LCRC Accounts System