September 2015 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Yearly Allocation Request for Comp_POP_Catalysts
by accounts＠lcrc.anl.gov 17 Sep '15

17 Sep '15

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Fanjie Kong Project Name: Comp_POP_Catalysts Division: CSE Project title: Computational studies of copper metalated catechol-containing porous organic polymers catalysts Associated funding: Catalysts Other Systems: None Science: The goal of this project is to explore, understand, and characterize, through modeling and simulations, the complex correlations between the structure, electronic features, and chemical reactivity of porous organic polymers (POPs) decorated with high densities of copper-(catecholate) groups. These discrete, monocatecholated complexes would be highly useful for applications such as gas storage,catalysis, or gas separations owing to their high specific surface area, good chemical stability and high structure control during synthesis. The central aim is to aid the ongoing experimental effort currently underway at CSE on synthesis of novel heterogeneous – primarily copper metalated catechol-containing POPs with superior activity and selectivity. Among the targeted catalytic transformations is selective oxidation of benzyl alcohol to benzaldehyde, which is a process of importance in various chemical technologies including those relevant to novel sources of energy. Project description: The planned computations will explore the structural, energetic, electronic, and catalytic functionality of copper incorporated catechol-containing POPs. Our studies will include the investigation of the structure and electronic structure, e.g. the charge state, along with the chemical functionality of active sites, composed of single or two copper atoms, incorporated onto catechol-containing POPs. The effects of ligands on these physical and chemical properties will also be explored. One of the candidates for catalytic transformations to be tested is the selective oxidation of benzyl alcohol to benzaldehyde. The activity and selectivity difference in the oxidation reaction of benzyl alcohol between single and two copper atoms containing catechol-containing POPs will be compared and explained, in correlation with the structural, charge and electronic structure analysis. The choices for the catechol-containing POPs and catalytic transformations are ma de based on the planned experimental investigations, some of which are currently undergoing. Within the planned studies, the issues that will be addressed are: 1) Activity and selectivity of single and two copper atoms containing catechol-containing POPs in the oxidation of benzyl alcohol. In this respect, computations have been partly performed for optimizing single and two copper atoms incorporated into catechol-containing POPs. These studies have been extended to single and two copper atoms containing catechol-containing POPs decorated with H2O, CH3OH and CH3CN as ligands. 2) Use of oxygen molecule to probe the oxygen activation of copper sites of single and two atoms supported on catechol-containing POPs. Here we plan to explore the most preferred adsorption form (molecular adsorption or dissociative adsorption), the most preferred adsorption conformation and the reaction pathway for adsorption. For this purpose some computations have been performed, where the potential energy surfaces are studied as a function of the distance between the mass center of oxygen molecule and copper for single copper catechol-containing POPs (the mass center of two copper atoms catechol-containing POPs). 3) Use of benzyl alcohol, in particular the concentration as a selectivity knob for selective oxidation of benzyl alcohol. We will particularly address the energetics of the benzyl alcohol adsorption as a function of benzyl alcohol concentration. 4) Our calculations on DFT-based assignment of infrared (IR) spectra of catecholate POP have shown an excellent agreement with the measured spectra. Hence, we aim to use the computed and the measured IR spectra as a tool for identification of the actual conformations of single and two copper atoms containing catechol-containing POPs. The computational studies of the structural, energetic, and electronic structure aspects will be performed using density functional theory. The packages that will be utilized include VASP, NWChem, CP2k and Crystal. Naturally, we will benefit from the efficient scaling of the codes and take advantage of it in order to explore systems with still increasing number of atoms. Thus in our project: 1) The study of the structural and electronic properties for single and two copper atoms containing catechol-containing POPs, and 2) the relation of these properties with the catalytic functionality, are research subjects which suit well with the high-performance infrastructure of the LCRC computational resources. Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 720000 Q1: 180000 Q2: 180000 Q3: 180000 Q4: 180000 Justification: The scaling efficiency on Fusion was tested using the code NWCHEM in geometry optimization of single copper catechol porous organic polymer, which contain a total of 95 atoms: 40 atoms of H (2 basis function per H), 49 atoms of C (14 basis function per C), 4 atoms of O (14 basis function per O) and 2 atoms of Cu (35 basis function per Cu); a total of 962 basis functions. Substantial speed-up is achieved with the increase in the number of cores: Cores Average SCF cycle (s) 96 1500 128 1000 Storage requirements: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for polarons
by accounts＠lcrc.anl.gov 17 Sep '15

17 Sep '15

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: John J. Low Project Name: polarons Division: CLS Project title: Polarons in Urania Associated funding: DOE Other Systems: Science: Safe nuclear power is important for the country’s energy security. The Fukushima Daiichi nuclear disaster in Japan demonstrated that little is known about how nuclear fuel materials behave during a meltdown of the core in a nuclear reactor. A recent review stated that “because of measurement difficulties and lack of calibration standards, properties of liquid UO2 that are important for reactor-safety assessments still have large uncertainties.” [Fink, J. Nucl. Materials, 279(2000)1]. A study of severe nuclear accidents by the French Atomic Energy Commission show that a better knowledge of “corium”, a lava-like mixture of primarily of molten fuel and its zirconium cladding, is fundamental to understanding and modeling this kind of accident. The objective of this research is to develop a predictive atomistic model of hot solid and molten nuclear systems. The lack of detailed experimental results obtained from costly and challenging experiments at extreme physical conditions makes advanced modelling techniques essential tools for gaining further insight into the material behavior. A recent study of the structure of molten uranium dioxide with X-Ray diffraction and molecular dynamics (MD) by Skinner, et al [Science, 346(2014)984] demonstrated that after fitting the parameters of a MD simulation to the X-Ray structure factors, excellent agreement was obtained between theory and experiment. While it is encouraging that a MD simulation can be fit to the long order of molten Urania, variable stoichiometry and polarons play an important role in the thermodynamics of urania near 3138K, the melting point of urania [Winter and MacInnes, J. Nucl. Mat. 137(1986)161]. These terms were not included in the MD simulation of Skinner, et al. We propose to develop a reactive force-field, to model variable stoichiometry and polarons in high temperature urania. This model will include the physics for an accurate description of urania, which can predict the structure, thermodynamics and properties of urania. Project description: The charge optimized many body (COMB) force-field of Li, et al. [J. Phys.: Condens. Matter 25(2013)50401 provides a fair description of the low temperature thermodynamics, polymorphs and defects of urania. Since the fitting database did not explicitly include the effect of excited states, the high temperature density, lambda transition and melting temperature were not correctly predicted. Polarons in urania involve the disproportionation of U(IV) to U(III) and U(V) and other oxidation states. We proposed to add these excited state atoms as species in the COMB force-field of and use the “Free Hopping Approximation” [Yakub, et al. Journal of Nuclear Materials 389 (2009) 119] to include the effect of polarons in molecular dynamics calculations in LAMMPS. We will need to run 500 1 nansecond trajectories of unit cells containing 10,000 atoms to span the relevant temperatures and compositions. We estimate that this will require 100,000 core hours. We propose to use hybrid density functional to predict the energy of polarons. These energies will be needed to parameterize the force-field for excited states. Hybrid functional density functional theory can be used to make accurate predictions of excited states of materials without an adjustable parameter for every element [Erhart, et al., Phys.Rev.B 90(2014) 035204]. Comparison of results from CP2K and ABINIT (and/or Dorado’s version of VASP) will be made. Gaussian basis sets, efficient implementation of hybrid functionals and excellent parallel scaling make CP2K an efficient choice for hybrid functionals. However, it is difficult to determine how basis set size and unit cell size in CP2K will influence the accuracy of predicted excitation energies. ABINIT (and Dorado’s version of VASP) include occupation matrix control which will be needed to converge to the desired electronic state [Dorado, et al. J. Phys.: Condens. Matter 25(2013)333201]. The results from the plane-wave code will serve as benchmarks for CP2K to determine what level of basis set is needed for accurate polaron energies. CP2K will be used to generate the data for the fitting database for the COMB potentials. The program GARField will be used to fit the COMB force-field to a database of calculated energies for urania polymorphs, defects and polarons [Jaramillo-Botero, et al., J. Chem. Theory Comp. 10(2014)1426]. The CPU time require to generate energies for polaron formation and fit the force-field will require 200,000 hours. There will be four project members in this project Marius Stan(NE), Alejandro Lopez-Bezanilla (MSD) and Laura Ratcliff (ALCF). Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 300000 Q1: 75000 Q2: 75000 Q3: 75000 Q4: 75000 Justification: Storage requirements: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for WEST
by accounts＠lcrc.anl.gov 17 Sep '15

17 Sep '15

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: John J. Low Project Name: WEST Division: MCS Project title: GW for the materials science community: extending transferability, benchmarking and addition of new productivity tools within the WEST code Associated funding: LDRD at ANL-MSD (PI), co-PI is staff at the UChicago’s Research Computing Center (RCC). MICCoM (DOE-funded computational center directed by G. Galli) Other Systems: Midway (RCC at UChicago) Science: The investigation of new materials that can extract, store and use energy in a more efficient way represents a key challenge of basic energy science research. First principle simulations based on Density Functional Theory (DFT) and Many-Body Perturbation Theory (MBPT) are expected to accelerate materials discovery while substantially lowering manufacturing costs. Recently we have released the open-source code WEST (www.west-code.org) which implements the DFT+MBPT scheme adopting several algorithmic and numerical advantages that are specifically targeted to tackle large scale simulations, proving a good scalability on >200k cores. In this project we propose to broaden the WEST user base and the number of users capable of using leadership computing resources by: i) establishing a publicly accessible database of results, ii) improving the transferability of data files across multiple platforms through the development of an interface with portable I/O libraries (e.g. H DF5 and NetCDF), and iii) developing pre- and post-processing tools to improve productivity with WEST. All codes used in this project and developed by the team are open-source, making them freely available to the scientific community of users. The findings of this project will be published in peer-reviewed journals, and the results of the simulations will be exported in the form of digital contents to be disseminated at national and international conferences, and in press articles and webpages. The search for new materials that can extract, convert and store energy with greater efficiency represents a grand challenge of basic energy sciences, with federal funding agencies providing continued support of this mission at the fundamental research level. In the field of materials science, researchers are striving to find non-toxic, disposable, cheap and durable materials that can be made more functional than those in current use. In particular, with the Materials Genome Initiative strategy (www.whitehouse.gov/mgi) first principle simulations based on density functional theory (DFT) are expected to accelerate discovery while substantially lowering manufacturing costs. Important quantities in the design of applications for photovoltaics, photocatalysis, and quantum information systems are computable physical properties like, band gaps, band edge positions, defect energy levels and absorption spectra. These properties are often poorly described by semilocal DFT. For more r eliably accurate physical descriptors it is typically required to compute these properties by combining DFT with Many-Body Perturbation Theory (MBPT), which requires the solution of the quasiparticle equation [1]. Along with an increased accuracy comes a higher computational cost, which has been the main limitation of these techniques to large-scale applications thus far. Project description: John Low is acting as temporary PI until Marco Govoni's LCRC account is fully activated. This allowed this project request to be submitted for a timely review by the allocation committee. Eventually John Low will most likely be removed as PI after Marco gets an LCRC account and becomes the formal PI for this project. Jonathon Skone will be a co-PI. There will be only two members of this project. Recently the PI(Marco Govoni) has released the open source code WEST (www.west-code.org) which implements the DFT+MBPT scheme adopting several algorithmic and numerical advantages that are specifically targeted to tackle large scale simulations. Details about the code can be found in Ref. [2], where scalability on >200k cores is discussed and systems of unprecedented size are simulated. In WEST the Dyson equation is solved in an approximate manner using iterative techniques that make use of matrix-free operations and density-based linear response theories [3]. A perturbative solution of the Dyson equation is obtained in two steps, the first of which involves an iterative diagonalization of the dielectric matrix via a Davidson diagonalization. At each step several linear systems are solved using the preconditioned conjugate gradient method to obtain the linear variation of the charge density, in the framework of Density Functional Perturbation Theory (DFPT). The second step i nvolves an expansion of the Green’s function and the screened Coulomb potential in a separable manner, followed by a frequency convolution via a Lanczos iterative algorithm. The iterative diagonalization of the dielectric matrix uses a deflation technique that does not require the calculation of already converged eigenvectors. Overall, the combination of recasting the solution of quasiparticle equations using mathematical techniques with the exploitation of massively parallel algorithmic implementations allows WEST to provide highly accurate physical descriptors for systems of unprecedented size (1000s of electrons or more). WEST uses as input the electronic structure evaluated with Density Functional Theory (DFT) codes such as Qbox [4] and Quantum Espresso [5], which solve the Kohn-Sham equations using the plane wave pseudopotential method. Code interoperability strongly calls for the adoption of a transferable I/O paradigm, which favors the creation of an easily convertible and architecture independent database. By interfacing WEST with parallel and transferable I/O libraries, eg. HDF or netCDF, we will solidify the use of a multi-platform workflow paradigm for production runs. Data transferability will also facilitate the ability to post-process data, obtained on remote resources, on local machines with tools under development. Finally, in order to provide a reference database of GW simulations, we propose to create a benchmark of WEST results, that will be published in a refereed journal article and made publicly accessible through the WEST website. >From the resources allocated within this project, we will broaden the WEST user base and the number of users capable of using leadership computing resources. This project supports time allocation on Argonne’s Laboratory Computing Research Center Blues cluster. The activity of this proposal will be coordinated with the materials simulation environment of MICCoM, the newly establish computational DOE center (ANL-PI G. Galli), and eventually be integrated into it. The project will also be interfaced with the research activity of the Galli group at the Institute for Molecular Engineering (http://galligroup.uchicago.edu) In the following we outline and describe in detail the three aims of this proposal. Aim #1: Establishing a publicly accessible database of GW results procured using WEST to serve as both a reference and a benchmark for the scientific community. The techniques unique to WEST, whereby the arduous task of converging the unoccupied manifold is eliminated, provide an improved state-of-the-art methodology for MBPT electronic structure calculations. These results may serve as a benchmark to other traditional MBPT codes where the accuracy is strongly dependent on the convergence of the unoccupied states and often times is approximated or extrapolated. We intend to build our database of WEST results starting from a recently proposed data set of 100 molecules (GW100) [6]. The previously published quasiparticle results compared several traditional GW codes, but there were several drawbacks in this reference data set for which we will improve upon by building a benchmark database of WEST results. The first issue with the aforementioned reference GW data set is that all results reported are for traditional GW methodologies that require the convergence of the unoccupied states. With WEST results the need for converging the unoccupied manifold of states is removed thus providing a superior reference point that even the traditional GW codes could take as a benchmark for establishing their convergence. Another issue lies with the DFT starting point reference to compute the QP corrections with a single step GW approach. Since the wavefunctions and the dielectric function (screened Coulomb interaction W) are not iteratively solved till self-consistency in a single step GW approach, the starting point upon which perturbative corrections are evaluated is of critical importance. The importance of the starting point, especially for systems where the dielectric screening and wavefunctions are markedly different depending on the starting point reference DFT is discussed in recent papers where WEST was used [7-10]. In the database of WEST results we aim at establishing GW results obtained with hybrid density functional reference starting points. We also intend to expand the set of reference data for 100 molecules to include a dozen or more molecules of relevance to the organic semiconducting industry, namely the set of variously branched polycyclic aromatic molecules (e.g. rubrene and pentacene) and charge transfer complexes such as tetracyanoquinodimethane--tetrathiafulvalene, for which presently no established reference GW results are available. The results of the benchmark will be published and serve as reference for contemporary and future GW calculations. All input and output files (inclusive of convergence tests) will be made available on a dedicated section of the WEST website. The large volume of meta-data will be stored at the Research Computing Center at the University of Chicago and served up through the WEST site. The data will provide a benchmark for future theoretical implementations and be inclusive in the set of reference test calculations used to ensure the fidelity of the WEST source code. The generated database will be used also by the WEST integrity software test suite known as the WEST Regression Test (WRT), which is comprised of a series of input tests, that are run on Midway at the RCC on a nightly basis checking the output generated from each test input file against the repository of verified correct output. The set of 100+ molecule quasiparticle computations with WEST requires approximately 1.25M hours of computing time. Aim #2: Improving transferability of data files generated and used by the massively parallel software WEST. We intend to further enhance the transferability of data files generated from parallel I/O in WEST through the implementation of HDF5 [11]. This aim requires the use of a set of diverse high performance computing architectures in order to test the transferability of data files generated from parallel I/O and will be addressed by initiating tasks either at ALCF or LCRC, then moving the initial data files to be restarted either at the LCRC or the RCC at the University of Chicago. We are requesting a total of 500k core hours of time at LCRC for the purpose of testing the transferability of data files generated by WEST at LCRC’s Blues cluster and transferred within Argonne to the ALCF’s Mira supercomputer or offsite to the Research Computing Center (RCC) cluster at the University of Chicago. In addition to making data transferability possible across diverse computer cluster architectures, this proposal serves to bolster the communication and collaborative efforts between the computational centers at Argonne National Laboratory (LCRC) and the University of Chicago (RCC). There are several steps involved in the procurement of results using WEST. Prior to even running WEST, the user needs to obtain a structurally converged geometry and a ground state electronic structure which may be obtained with other scientific software such as LAMMPS, Qbox, or Quantum Espresso etc. This step, due to lower degree of complexity and computational cost may be run on a mid-sized cluster such as (LCRC or RCC at the University of Chicago). The data files generated for the ground state electronic structure would then need to be transferable if the user is to then use this information to initiate a WEST calculation on a larger sized supercomputer such as the ALCF’s Mira (or the new upcoming ALCF machines Theta and Aurora). It is also often the case that a resource allocation may expire or become depleted and the user must move their incomplete WEST job to another resource center. In order for the job to restart from these data files generated on a different archit ecture a transferable I/O data structure is needed. Furthermore, once results are obtained using WEST, often times it is necessary to run postprocessing tools to further analyze the data and/or make meaningful plots. This is a much less computationally demanding task that the user can achieve on a modest desktop or laptop computer, but in order to do so the data files need to be generated from transferable parallel I/O to be further manipulated. This aim will also result in a technical paper describing the parallel I/O structure that will be published in a peer-reviewed journal. We also expect this project to bolster the relationship between ANL-LCRC/ALCF and the University of Chicago Research Computing Center. Aim #3: Pre- and post-processing tools to improve productivity with WEST. The creation and testing of new pre- and post-processing tools will provide increased user productivity and extend the reach of systems suitable for analysis with WEST. Of the existing post-processing tools (e.g. Wannier center analysis, Spectral function plot, STM imaging), improvement in the parallelization of these routines will provide for improved efficiency of resource use and a quicker turnaround time in the data analysis of WEST results. The new post-processing tools we intend to implement are those which will have an immediate impact on the interpretation and data visualization of WEST results. These new post-processing tools include a set of python modules to graphically plot the quasiparticle corrected photoelectron spectra, the associated lifetimes, and visualization plots of the dielectric function. To expand the scope of applicable systems that can be treated with WEST we will implement the routines necessary to handle more general lattice systems, making it possible to then handle, for instance, monoclinic crystal systems--typical of many semiconducting organic molecular crystals. To carry out the needed testing of the implementation of arbitrary unit cell input, the new post-processing tools for plotting quasiparticle photoelectron spectra / lifetimes, and the parallelization of the existing post-processing tools, an additional 250k hours of computing time is required. Justification of the number of requested core hours: In order to create the WEST GW database of results for a set of 100+ molecules we need a total of 1,250,000 h. In order to test the implementation of HDF5 with WEST on large systems of unprecedented size such as nanostructures and molecular crystals, we need to run transferability tests and production runs that will require approximately 500000 h. In order to test the the new pre- and post-processing set of productivity tools in WEST we need approximately 250000 h. The total number of hours requested is 2 million CPU hours. Team qualification: Marco Govoni is a research associate at the Institute for Molecular Engineering in the Materials Science Division of Argonne National Laboratory. He also has a joint visiting scientist appointment at the University of Chicago. He holds a PhD in Nanoscience and Nanotechnology from the University of Modena and Reggio Emilia, Italy. He was postdoctoral scholar at the University of California Davis (UC Davis). His main interest is modeling and understanding materials for energy research. He is the team coordinator and director of software engineering of the massively parallel many-body perturbation code WEST. He has collaborated with the Supercomputing Application and Innovation Department (SCAI) at CINECA, Italy in the past where in 2012 he was awarded an ISCRA scholarship for petascale software development. His experience in high performance computation and his experience as a software developer guarantees an efficient, productive and solid management of the allocated resources . The quality of the applicant’s HPC-related work is supported by publications in high-level journals, e.g. Nature Photonics and Journal of the American Chemical Society [12-13]. He will provide an optimal management of the resources allocated for MBPT simulations. His source of funding comes from projects led by Prof. G. Galli at ANL and UChicago. He will be heading the simulations and the scientific discussion with the other team member. Jonathan H. Skone is a staff member of the Research Computing Center at the University of Chicago, where his primary responsibility is to provide HPC solutions to the Institute for Molecular Engineering. He previously held a postdoctoral appointment within the Materials Science Division at Argonne National Laboratory where he pursued projects related to water catalysis with oxides and methodological development for condensed matter systems. He holds a PhD in Chemistry from Pennsylvania State University. His experience, from past projects carried out at as a postdoc at both Brookhaven National Laboratory and UC Davis, include the modeling of materials and molecules for electrocatalysis and photocatalysis. With the PI, he authored publications in hybrid functionals development [7-9], whose applicability to large scale modeling of aqueous electrolytes will be explored in the recently awarded ALCC project. The inclusion of Jonathan in the team is motivated by his expertise on sim ulations with the WEST code. He will serve as the point of contact for maintaining the database of benchmarked systems and will develop the necessary routines in order to interface WEST with the targeted I/O libraries (HDF5 and NetCDF). Being a research consultant for the Institute for Molecular Engineering, he will make sure that the project is well embedded in the newly established engineering research center, whose mission is to build a partnership between the University of Chicago and ANL which will be bolstered by this increased interaction and scientific collaboration. References: [1] G. Onida, L. Reining, and Angel Rubio, Review of Modern Physics 74, 601 (2002). [2] M. Govoni, and G. Galli, J. Chem. Theory Comput. 11, 2680 (2015). [3] Y. Ping, D. Rocca, and G. Galli, Chem. Soc. Rev. 42, 2437 (2013). [4] https://qboxcode.org [5] P. Giannozzi et al., J. Phys.: Condens. Matter 21, 395502 (2009). [6] http://th.fhi-berlin.mpg.de/site/uploads/Publications/GW100-1_20150518.pdf [7] J. H. Skone, M. Govoni, and G. Galli, Phys. Rev. B 89, (2014). [8] J. H. Skone, M. Govoni, and G. Galli, “Using Dielectric Properties to Design Nonempirical Hybrid Functionals for Accurate Electronic Structure of Molecules and Materials.” Phys. Rev. B To be submitted [9] A. Gaiduk, M. Govoni, J. H. Skone, and G. Galli, “The Photoelectron Spectrum and absolute band positions of ions in water using many-body perturbation theory with various DFT reference starting points.” To be submitted [10] H. Seo, M. Govoni, and G. Galli, “Design of defect spins in piezoelectric aluminum nitride for solid-state hybrid quantum technologies.” Submitted [11] https://www.hdfgroup.org/HDF5/ [12] M. Govoni, I. Marri, and S. Ossicini, Nature Photonics 6, 672 (2012). [13] I. Marri, M. Govoni, and S. Ossicini, J. Am. Chem. Soc. 136, 13257 (2014). Industry partnership: Project URL: http://www.west-code.org/ Current FY Hours Used: undetermined amount New FY Requested allocation: 2000000 Q1: 500000 Q2: 500000 Q3: 500000 Q4: 500000 Justification: Software Readiness: We report in the following, the details of the WEST code, which will be used extensively in this project. WEST (Without Empty STates). Accurate electronic structures will be computed using MBPT. Quasiparticle corrections will be computed using the G0W0 approach where the electronic self-energy is solved using iterative techniques developed by the PI [2]. The details of these techniques, that do not require the explicit calculation of empty electronic states and fit the inversion of large dielectric matrices (typical problems of large scale systems), are given in a recent review paper [3]. The screened Coulomb interaction W is obtained in the random phase approximation by representing the dielectric screening with a projective eigendecomposition basis set. The iterative diagonalization of the screening with the Davidson’s algorithm is started with random potentials and then converged in 4-6 steps. At every iteration, each eigenpotential gives rise to a stand-alone Sternheimer problem where the perturbed density is computed. Because each perturbation can be treated independently, this part of the algorithm i s embarrassingly parallel. GW self-energies are then obtained using a combined Lanczos-Deformation Contour approach, analogously parallelized. WEST is an open-source software (www.west-code.org) which is maintained by the PI. WEST is interfaced with QuantumEspresso (http://www.quantum-espresso.org) which is also open-source and already installed in the targeted HPC architecture. The code uses a hybrid OpenMP-MPI parallelization scheme and, although self-contained, can take advantage of highly optimized libraries like BLAS, LAPACK, SCALAPACK, FFTW3, ESSL, PESSL, GDLib and GSL. WEST implements a multiple level of MPI communicators providing massive parallelism even for large systems. West have demonstrated close to linear parallel scaling up to 3000 cores with fair parallel scaling to 6000 cores on the BG/Q Mira at ANL-ALCF for a liquid/solid interface model containing 492 atoms and 1560 valence electrons[2]. Storage requirements: Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] 15 more new projects
by John Blaas 16 Sep '15

16 Sep '15

Going to grant all of the following projects with 50k initial allocation tomorrow morning if no one has any comments. The following projects are: CataAgCl CCCTMO CoC_tersoff Comp_POP_Catalysts FF-opt HEHV_modeling iro-catal IVwire Lanthanide_FF Li-air_phase Li-S_Trans Meso_Nuclear PhotoCu2O polarons PrtnAdp_CG Superlubricity_ND_2D VO2_thermal WEST Waiting off on BrewerDefectForm since the proposal made it sound like this was the proprietary project from Brewer, unless things have been worked out with legal of course to give this project the greenlight? Thanks, JB On Tue, Sep 15, 2015 at 1:57 PM, Bair, Raymond A. <rbair(a)anl.gov> wrote: > We suddenly have 15 requests for new projects this week, all requesting > time in Q4. > > Earlier in the month we capped new requests at 100K. That should be less > now, perhaps 75K or 50K? > > Also, each of these 15 will need to be reminded to turn in a FY16 request > too. I suppose they need to wait until their FY15 time is approved, so > time is if the essence. > > Ray > > _______________________________________________ > allocations-admins mailing list > allocations-admins(a)lcrc.anl.gov > https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Yearly Allocation Request for Electric_Interface
by accounts＠lcrc.anl.gov 16 Sep '15

16 Sep '15

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: John J. Low Project Name: Electric_Interface Division: MSD Project title: Electrified Interfaces from First-Principle Studies Associated funding: DOE BES-CEES EFRC, DOE BES-JCESR Other Systems: Science: Electrified Interface of solid and solid/liquid electrolytes in electrochemical system is an important component that of great interest to electric energy storage research at Argonne (i.e. BES-JCESR, BES-CEES-EFRC, DOE-EERE), and also among the scientific community. Within this wide area of research problems, the effect of external bias voltage on the interface at the atomistic scale is the basis of our fundamental understanding, design and application in this field. To include the external bias voltage in atomistic simulation based on first-principle study (DFT), the theoretical treatment with the application of a finite homogeneous electric field proposed by Nunes et. al. [1] is a reasonable and practical method based on DFT calculations for the fundamental studies. Within this renew proposal, the key parts of the proposed study will consists of two independent yet related topics: (I) the electrified interface of organic electrolyte/oxide, (II) electrified interface of organic electrolyte/CrSi2. For the case study of electrified interface of organic electrolyte/oxide, our interest will continue to focus on oxide solid-phase instability at external electric potential that exemplified by metal dissolution from manganese (Mn)-based oxide cathode surface that found in next generation Li-ion battery system [2]. To go in parallel with our experimental colleagues, the modeled system will conitue to focus on the spinel LiMn2O4 (LMO)/ GenII electrolyte (i.e. 1.0 LiPF6 in ethylene carbonate: ethyl methyl carbonate electrolyte with 3:7 ratio by weight) developed at ANL. In the coming FY2016, the primary study will continue be Mn desolvation, dissolution and diffusion at the cathode interface as we planned on FY2015, but will only focus on [100] LMO. For the case study of electrified interface of organic electrolyte/CrSi2, our interest is impired by the experimental work by Fister et. el. [3]. A layer of Li cations have been observed by Fister et al. at the interface between CrSi2 and a electrolyte contain LiPF6, ethylene carbonate and propylene carbonate at well define voltages. [3] We will use to a combination of quantum mechanical and force-field calculation to model Li cations at this electrified interface and compare them to experimental results. Project description: Experiments Planned: The project can be divided into two related problems: 1) Electrified Interfaces of spinel [100] LMO/electrolyte According to our knowledge, it is found that the surfaces of [100] LMO is susceptible to dissolution than [111] LMO surface. Therefore this surface appears to be a logical modeled system to focus on. To address the problems that related to our experimental colleagues findings, the simulation of Mn dissolvation, dissolution and diffusion into electrolyte interfaces will be performed with and without the water molecules (i.e. moisture that will cause side-effect) in the simulation. 2) Electrified Interfaces of CrSi2/electrolyte We will used the OPLS-AA forcefield and LAMMPS to generate a starting configuration for the bilayer at the CrSi2/electrolyte. Molecular dynamics is the most efficient way to preequilibrate, allowing each ion to attain proper coordination and a low energy distribution in the bilayer. The bilayer in carbonates is about 30Å wide. We plan to carry out calculations to span the voltages where the layer of Li cations were observed. The first pass will involve a molecular dynamics followed by geometry optimization with quantum mechanics. This should yield a qualitative correct description of the electrified CrSi2/electrolyte interface. If sufficient CPU time is available we will carry out ab initio molecular dynamics trajectories on the most interesting optimized geometries. Computational Methods and Software The density functional theory simulation code OPENMX, VASP and Quantum Espresso will be employed in most of the work. Visualization The public domain graphics packages VMD, VESTA, XCRYSDEN and PWGUI are convenient for depicting three dimensional atomic configurations, making movies of dynamical processes and setting up calculations. Scalability Issues VASP is well known to have scalability issues. Although it is very efficient code for models contains 100s of atoms, the parallel scalability for systems of this size is only 100s of cores. Although better parallel scalability can be obtained for larger models, the size scalablity becomes a major issue as O(N3) terms become a bottleneck in the simulation. The scalability of Quantum Espresso is similar to that of VASP. OpenMX is an O(N) code with good parallel scalability. A recent publication by Duy and Osaki [4] has demonstrated the excellent parallel scalability of OpenMX to over a 100K cores on the K computer. Calculation size The smallest cells for which the effects of interest can realistically be simulated for [100] LMO/electrolyte AIMD simulation will contain about four - five hundred atoms, with about 200 atom slab of LMO, and 200 - 300 atoms of electrolyte that represent ESM screening media in OPENMX. All numerical shortcuts (gamma-point only, soft pseudopotentials) will be exploited. We expect to run on 256 processors. Of the calculations proposed here, the AIMD simulations (item 1) will have the largest CPU-time requirements per unit MD-time step of 1 fs. The AIMD simulations performed in NVT-ensemble simulations may require about 1 processor-hr per time step, for a cell size of about 500 atoms (using perhaps 256 processors). Obtaining a reasonably good MD statistics calculation may require of the order of 200 k processor hrs. Thus for two separate sample simulation (i.e. with and without H2O molecule), the total 400k core-hour is needed in this problem. Our model includes a three-layer slab of CrSi2 about 15Å wide containing 108 atoms (Cr36O72), a 30Å layer of electrolyte containing 6 LiPF6 molecules and 60 ethylene carbonate molecules. We estimate a ten quantum mechanical geometry optimizations with different charges on the CrSi2 slab. Each geometry optimization will require 3,000 core-hours. Ten geometry optimizations will require 30,000 core hours. The voltages from these preliminary calculations will help us select three geometries for ab initio molecular dynamics (AIMD). A picosecond of AIMD will require 60,000 core-hours. Three trajectories at different levels of charging will require 180,000 core-hours. The voltages from these AIMD simulations will suggest an additional three AIMD simulations with different levels of charging to attain the desire voltage in the simulation. The total amount of time for the geometry optimization and AIMD simulations of CrSi2/Li6(PF6)6EC6 will require 400,000 core-hours. Thus, requirements for the entire project are thought to be of order 800,000 core-hours. References: 1. R. W. Nunes and X. Gonze, Phys. Rev. B 63, 155107 (2001). 2. J. Lu et. al. Nature Communications 5, 5693 (2014). 3. T. Fister et. al. J. Phys. Chem. C 116(2012)22341. 4. T. Ohwaki, M. Otani, T. Ikeshoji, T. Ozaki, J. Chem. Phys. 136, 134101 (2012); T.V.T. Duy, T. Osaki, Comp. Phys. Comm. 185, 777 (2014). Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 800000 Q1: 200000 Q2: 200000 Q3: 200000 Q4: 200000 Justification: We have been testing the beta version of VASP on GPUs. We expect that the GPU version of VASP will be three times faster than the current version. This should increase the efficiency of our VASP calculations. See http://www.mcs.anl.gov/~jlow/VASP/Scaling%20data%20for%20VASP%20on%20Blues.… for details on the parallel scaling of VASP. See http://www.sciencedirect.com/science/article/pii/S0010465513004013 for details for the parallel scaling of OPENMX. See http://training.uhem.itu.edu.tr/docs/18hazirannano/PW-III-para.pdf for details on the parallel scaling of Quantum Espresso. Storage requirements: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for homme_moab
by accounts＠lcrc.anl.gov 16 Sep '15

16 Sep '15

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Iulian Grindeanu Project Name: homme_moab Division: MCS Project title: Scalable Advection Methods for Multi Tracers in Climate Codes Associated funding: SciDAC FASTMATH, ACES4BGC, DOE Other Systems: VESTA / MOAB -scalability , have trouble compiling there Science: One of the important goals of ACES4BGC project (Applying Computationally Efficient Schemes for BioGeochemical Cycles) is to implement and optimize new computationally efficient advection algorithms for large number of tracer species. This work demonstrates a framework that uses intersection algorithms developed in MOAB (Mesh Oriented datABase), linked with a HOMME dynamical core driver. Project description: Several studies were conducted with respect to intersection scalability, initial results were presented at 2015 SIAM-CSE Conference. Studies need to continue, as several scalability bottlenecks were identified: use HDF5 for parallel IO scales satisfactory on blues system, up to 256 nodes / 16 tasks per node; writing/reading scalability was demonstrated for a file with 6 billion entities, of which about 3 billion vertices and 3 billion hexahedrons. The generated mesh file uses 300Gb of allocated disk space. - some memory leaks need to be found / fixed, as it is found that after several thousands intersection iterations the system runs out of memory. I am planning to use some profiling tools, to help identify where the problem lies on. - the setup of the problem is rather cumbersome, HOMME code is written in Fortran 90, it has various dependencies, mostly HDF5, netcdf and pnetcdf, in the configuration used, and a cmake based build system; The new advection transport method is implemented as part of MOAB, in C++, and it uses a scalable mesh intersection algorithm, for the eulerian-lagrangian approach. - to communicate / locate intersection meshes on various mesh partitions, MOAB parallel infrastructure is used heavily; At the same time, HOMME already has a parallel infrastructure, but it is limited to communication within one layer of ghost elements; MOAB infrastructure allows arbitrary communication, but the effectiveness and advantages of this more complicated approach still needs to be demonstrated. MOAB approach would allow running the problem with CFL numbers larger than 1, although we need to demonstrate that this approach is theoretically sound. Industry partnership: Project URL: http://www.aces4bgc.org/ Current FY Hours Used: undetermined amount New FY Requested allocation: 40000 Q1: 10000 Q2: 10000 Q3: 10000 Q4: 10000 Justification: Storage requirements: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: Li-air_phase
by accounts＠lcrc.anl.gov 15 Sep '15

15 Sep '15

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Linyun Liang Applicant's institution: ANL Applicant's division: MCS Project Name: Li-air_phase Project title: Atomistically Informed Mesoscale Modeling for Advanced Electrical Energy Storage Systems Associated funding: LDRD Other Systems: Science: We propose to develop a multi-scale simulation approach that can be used to understand and optimize the design of cathode materials that used in Li-air batteries for future applications. The integrated approach provides a connection from the cathode microstructure-level to the electrochemical performance cell-level. This integrated approach could potentially quantify the time-dependent evolution of complex three-dimensional cathode microstructure during the electrochemical processes in an operating Li-air battery, and in turn to feedback to battery performance by offering the opportunity to improve the fundamental understanding of the electrode microstructures at the mesoscale level through the phase-field simulation. Project description: Besides the proper utilization of electrolytes, the design and controlled growth of the cathode structure and morphology evolution represent a major technology challenge in the development of Li-air batteries, which dictates the most part of the cell energy density and efficiency, and contributes to most of its voltage overpotentials. In a porous air cathode, Li ions and dissolved oxygen move through the electrolyte and react with transferred electrons at the cathode surface to form the Li2O2. In this project, we will develop a multi-scale model to study the nucleation and growth of discharge product (Li2O2, LiO2, etc.) in a three-dimensional porous carbon cathode in Li-air batteries that based on our previous studies [1, 2]. These will integrating a phase-field model and an electrochemical model, which span length scales from the mesoscale to the macroscale with the inputs of experimental parameters and atomistic modeling based on Density Functional Theory (DFT). The three-dimensional porous cathode structure is first generated by using the phase field approach. The porosity, pore size and its distribution, tortuosity, and surface area are the very important structural properties of cathode that can be measured by the experiment through the collaboration with battery team at Argonne. Then during the discharge process, the discharge products that nucleated on the carbon surface and their growth evolution, which it will eventually decrease the porosity and block the mass transport of solvated oxygen and electrolytes. Similarly these reaction parameters can also be obtained through the experiments. At the cell scale, with the diffusion of reactant species such as the oxygen, Li+ ions and electrolyte inside the pores, and also the flow of electrons transport in the electrode, these couple electrochemical reactions will be related to the cathode microstructure evolution and the cell’s performance in phase-field model. For some of the fun damental reaction, e.g. initial nucleation of reactant species (Li+, O2-, electrolyte molecules, and LixOy species or aggregates) on carbon surfaces, the reaction and interfacial energies will be estimated based on DFT calculations (using VASP and Gauusian09 code). For this part of calculations, we will need 100,000 core-hour in order to complete our needed parameters for the phase-field model. In our phase field model, we have at least eight highly non-linear, coupled partial differential equations which must be solved numerically. In this work, the three-dimension and large size of cathode size will significantly increase the cost of the simulation time. To solve this problem, we will use our in-house developed FFTW code to solve the problem. We will need 200,000 core-hour to debug and test the algorithm and code. Thus by make use of Fusion and Blues computing resources will greatly accelerated the code development for our future exploration of 3D systems with complicated morphology. To include the other fundamental atomistic features at small length scale investigations, the input parameters such as the surface energy, reaction rate constant, diffusivities, and solubility, etc. will then be obtained either through the literature search and experimental supports through Argonne co-workers. In general, the required CPU time for carbon structural optimizations will be (8 cores/node * 8 nodes * 20 hours of run time * 100 different runs) 128,000 core-hours and for diffusion calculations is (8 cores/node * 8 nodes * 20 hours of run time * 100 different runs)=128,000 core-hours. The total time requested including some test study on large system is about 300,000 core hours. In conclusion, the total core-hour requested for this project throughout the FY2016 will be 600,000 core-hour. Publication: 1) M.J. Welland, K. C. Lau, P.C. Redfern, L. Liang, D. Zhai, D. Wolf, L.A. Curtiss, An atomistically informed mesoscale model for growth and coarsening during discharge in lithium-oxygen batteries, under review in J. Chem. Phys. 2) Linyun Liang, K. C. Lau, M. J. Welland, M. Stan, M. Anitescu, D. Wolf, and L. A. Curtiss, Mesoscale Model for the LiO2-Li2O2 Transformation in a Lithium-Oxygen Battery, in preparation Industry partnership: Project URL: Requested allocation: 600000 Q1: 0 Q2: 0 Q3: 0 Q4: 600000 Justification: A 10*10*10 nm3 system needs 8 node x 8 core x 1 hour for a single calculation. If we go to 50 * 20 * 20 nm3 system needs 8 node x 8 core x 20 hours for a single calculation. We will need at least 300 different runs. Thus, the total core-hours requested will be 600,000 core-hours. Storage requirements: 1TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: WEST
by accounts＠lcrc.anl.gov 15 Sep '15

15 Sep '15

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: John J. Low Applicant's institution: ANL Applicant's division: MCS Project Name: WEST Project title: GW for the materials science community: extending transferability, benchmarking and addition of new productivity tools within the WEST code Associated funding: LDRD at ANL-MSD (PI), co-PI is staff at the UChicago’s Research Computing Center (RCC). MICCoM (DOE-funded computational center directed by G. Galli) Other Systems: Midway (RCC at UChicago) Science: The investigation of new materials that can extract, store and use energy in a more efficient way represents a key challenge of basic energy science research. First principle simulations based on Density Functional Theory (DFT) and Many-Body Perturbation Theory (MBPT) are expected to accelerate materials discovery while substantially lowering manufacturing costs. Recently we have released the open-source code WEST (www.west-code.org) which implements the DFT+MBPT scheme adopting several algorithmic and numerical advantages that are specifically targeted to tackle large scale simulations, proving a good scalability on >200k cores. In this project we propose to broaden the WEST user base and the number of users capable of using leadership computing resources by: i) establishing a publicly accessible database of results, ii) improving the transferability of data files across multiple platforms through the development of an interface with portable I/O l ibraries (e.g. HDF5 and NetCDF), and iii) developing pre- and post-processing tools to improve productivity with WEST. All codes used in this project and developed by the team are open-source, making them freely available to the scientific community of users. The findings of this project will be published in peer-reviewed journals, and the results of the simulations will be exported in the form of digital contents to be disseminated at national and international conferences, and in press articles and webpages. The search for new materials that can extract, convert and store energy with greater efficiency represents a grand challenge of basic energy sciences, with federal funding agencies providing continued support of this mission at the fundamental research level. In the field of materials science, researchers are striving to find non-toxic, disposable, cheap and durable materials that can be made more functional than those in current use. In particular, with the Materials Genome Initiative strategy (www.whitehouse.gov/mgi) first principle simulations based on density functional theory (DFT) are expected to accelerate discovery while substantially lowering manufacturing costs. Important quantities in the design of applications for photovoltaics, photocatalysis, and quantum information systems are computable physical properties like, band gaps, band edge positions, defect energy levels and absorption spectra. These properties are often poorly described by semilocal DFT. For more r eliably accurate physical descriptors it is typically required to compute these properties by combining DFT with Many-Body Perturbation Theory (MBPT), which requires the solution of the quasiparticle equation [1]. Along with an increased accuracy comes a higher computational cost, which has been the main limitation of these techniques to large-scale applications thus far. Project description: John Low is acting as temporary PI until Marco Govoni's 593 has been completely processed. This allowed this project request to be submitted for a timely review by the allocation committee. Eventually John Low will most likely be removed as PI after Marco gets an LCRC account and becomes the formal PI for this project. Jonathon Skone will be a co-PI. There will be only two members of this project. Recently the PI(Marco Govoni) has released the open source code WEST (www.west-code.org) which implements the DFT+MBPT scheme adopting several algorithmic and numerical advantages that are specifically targeted to tackle large scale simulations. Details about the code can be found in Ref. [2], where scalability on >200k cores is discussed and systems of unprecedented size are simulated. In WEST the Dyson equation is solved in an approximate manner using iterative techniques that make use of matrix-free operations and density-based linear response theories [3]. A perturbative solution of the Dyson equation is obtained in two steps, the first of which involves an iterative diagonalization of the dielectric matrix via a Davidson diagonalization. At each step several linear systems are solved using the preconditioned conjugate gradient method to obtain the linear variation of the charge density, in the framework of Density Functional Perturbation Theory (DFPT). The second step i nvolves an expansion of the Green’s function and the screened Coulomb potential in a separable manner, followed by a frequency convolution via a Lanczos iterative algorithm. The iterative diagonalization of the dielectric matrix uses a deflation technique that does not require the calculation of already converged eigenvectors. Overall, the combination of recasting the solution of quasiparticle equations using mathematical techniques with the exploitation of massively parallel algorithmic implementations allows WEST to provide highly accurate physical descriptors for systems of unprecedented size (1000s of electrons or more). WEST uses as input the electronic structure evaluated with Density Functional Theory (DFT) codes such as Qbox [4] and Quantum Espresso [5], which solve the Kohn-Sham equations using the plane wave pseudopotential method. Code interoperability strongly calls for the adoption of a transferable I/O paradigm, which favors the creation of an easily convertible and architecture independent database. By interfacing WEST with parallel and transferable I/O libraries, eg. HDF or netCDF, we will solidify the use of a multi-platform workflow paradigm for production runs. Data transferability will also facilitate the ability to post-process data, obtained on remote resources, on local machines with tools under development. Finally, in order to provide a reference database of GW simulations, we propose to create a benchmark of WEST results, that will be published in a refereed journal article and made publicly accessible through the WEST website. >From the resources allocated within this project, we will broaden the WEST user base and the number of users capable of using leadership computing resources. This project supports time allocation on Argonne’s Laboratory Computing Research Center Blues cluster. The activity of this proposal will be coordinated with the materials simulation environment of MICCoM, the newly establish computational DOE center (ANL-PI G. Galli), and eventually be integrated into it. The project will also be interfaced with the research activity of the Galli group at the Institute for Molecular Engineering (http://galligroup.uchicago.edu) In the following we outline and describe in detail the three aims of this proposal. Aim #1: Establishing a publicly accessible database of GW results procured using WEST to serve as both a reference and a benchmark for the scientific community. The techniques unique to WEST, whereby the arduous task of converging the unoccupied manifold is eliminated, provide an improved state-of-the-art methodology for MBPT electronic structure calculations. These results may serve as a benchmark to other traditional MBPT codes where the accuracy is strongly dependent on the convergence of the unoccupied states and often times is approximated or extrapolated. We intend to build our database of WEST results starting from a recently proposed data set of 100 molecules (GW100) [6]. The previously published quasiparticle results compared several traditional GW codes, but there were several drawbacks in this reference data set for which we will improve upon by building a benchmark database of WEST results. The first issue with the aforementioned reference GW data set is that all results reported are for traditional GW methodologies that require the convergence of the unoccupied states. With WEST results the need for converging the unoccupied manifold of states is removed thus providing a superior reference point that even the traditional GW codes could take as a benchmark for establishing their convergence. Another issue lies with the DFT starting point reference to compute the QP corrections with a single step GW approach. Since the wavefunctions and the dielectric function (screened Coulomb interaction W) are not iteratively solved till self-consistency in a single step GW approach, the starting point upon which perturbative corrections are evaluated is of critical importance. The importance of the starting point, especially for systems where the dielectric screening and wavefunctions are markedly different depending on the starting point reference DFT is discussed in recent papers where WEST was used [7-10]. In the database of WEST results we aim at establishing GW results obtained with hybrid density functional reference starting points. We also intend to expand the set of reference data for 100 molecules to include a dozen or more molecules of relevance to the organic semiconducting industry, namely the set of variously branched polycyclic aromatic molecules (e.g. rubrene and pentacene) and charge transfer complexes such as tetracyanoquinodimethane--tetrathiafulvalene, for which presently no established reference GW results are available. The results of the benchmark will be published and serve as reference for contemporary and future GW calculations. All input and output files (inclusive of convergence tests) will be made available on a dedicated section of the WEST website. The large volume of meta-data will be stored at the Research Computing Center at the University of Chicago and served up through the WEST site. The data will provide a benchmark for future theoretical implementations and be inclusive in the set of reference test calculations used to ensure the fidelity of the WEST source code. The generated database will be used also by the WEST integrity software test suite known as the WEST Regression Test (WRT), which is comprised of a series of input tests, that are run on Midway at the RCC on a nightly basis checking the output generated from each test input file against the repository of verified correct output. The set of 100+ molecule quasiparticle computations with WEST requires approximately 1.25M hours of computing time. Aim #2: Improving transferability of data files generated and used by the massively parallel software WEST. We intend to further enhance the transferability of data files generated from parallel I/O in WEST through the implementation of HDF5 [11]. This aim requires the use of a set of diverse high performance computing architectures in order to test the transferability of data files generated from parallel I/O and will be addressed by initiating tasks either at ALCF or LCRC, then moving the initial data files to be restarted either at the LCRC or the RCC at the University of Chicago. We are requesting a total of 500k core hours of time at LCRC for the purpose of testing the transferability of data files generated by WEST at LCRC’s Blues cluster and transferred within Argonne to the ALCF’s Mira supercomputer or offsite to the Research Computing Center (RCC) cluster at the University of Chicago. In addition to making data transferability possible across diverse computer cluster architectures, this proposal serves to bolster the communication and collaborative efforts between the computational centers at Argonne National Laboratory (LCRC) and the University of Chicago (RCC). There are several steps involved in the procurement of results using WEST. Prior to even running WEST, the user needs to obtain a structurally converged geometry and a ground state electronic structure which may be obtained with other scientific software such as LAMMPS, Qbox, or Quantum Espresso etc. This step, due to lower degree of complexity and computational cost may be run on a mid-sized cluster such as (LCRC or RCC at the University of Chicago). The data files generated for the ground state electronic structure would then need to be transferable if the user is to then use this information to initiate a WEST calculation on a larger sized supercomputer such as the ALCF’s Mira (or the new upcoming ALCF machines Theta and Aurora). It is also often the case that a resource allocation may expire or become depleted and the user must move their incomplete WEST job to another resource center. In order for the job to restart from these data files generated on a different archit ecture a transferable I/O data structure is needed. Furthermore, once results are obtained using WEST, often times it is necessary to run postprocessing tools to further analyze the data and/or make meaningful plots. This is a much less computationally demanding task that the user can achieve on a modest desktop or laptop computer, but in order to do so the data files need to be generated from transferable parallel I/O to be further manipulated. This aim will also result in a technical paper describing the parallel I/O structure that will be published in a peer-reviewed journal. We also expect this project to bolster the relationship between ANL-LCRC/ALCF and the University of Chicago Research Computing Center. Aim #3: Pre- and post-processing tools to improve productivity with WEST. The creation and testing of new pre- and post-processing tools will provide increased user productivity and extend the reach of systems suitable for analysis with WEST. Of the existing post-processing tools (e.g. Wannier center analysis, Spectral function plot, STM imaging), improvement in the parallelization of these routines will provide for improved efficiency of resource use and a quicker turnaround time in the data analysis of WEST results. The new post-processing tools we intend to implement are those which will have an immediate impact on the interpretation and data visualization of WEST results. These new post-processing tools include a set of python modules to graphically plot the quasiparticle corrected photoelectron spectra, the associated lifetimes, and visualization plots of the dielectric function. To expand the scope of applicable systems that can be treated with WEST we will implement the routines necessary to handle more general lattice systems, making it possible to then handle, for instance, monoclinic crystal systems--typical of many semiconducting organic molecular crystals. To carry out the needed testing of the implementation of arbitrary unit cell input, the new post-processing tools for plotting quasiparticle photoelectron spectra / lifetimes, and the parallelization of the existing post-processing tools, an additional 250k hours of computing time is required. Software Readiness: We report in the following, the details of the WEST code, which will be used extensively in this project. WEST (Without Empty STates). Accurate electronic structures will be computed using MBPT. Quasiparticle corrections will be computed using the G0W0 approach where the electronic self-energy is solved using iterative techniques developed by the PI [2]. The details of these techniques, that do not require the explicit calculation of empty electronic states and fit the inversion of large dielectric matrices (typical problems of large scale systems), are given in a recent review paper [3]. The screened Coulomb interaction W is obtained in the random phase approximation by representing the dielectric screening with a projective eigendecomposition basis set. The iterative diagonalization of the screening with the Davidson’s algorithm is started with random potentials and then converged in 4-6 steps. At every iteration, each eigenpotential gives rise to a stand-alone Sternheimer problem where the perturbed density is computed. Because each perturbation can be treated independently, this part of the algorithm i s embarrassingly parallel. GW self-energies are then obtained using a combined Lanczos-Deformation Contour approach, analogously parallelized. WEST is an open-source software (www.west-code.org) which is maintained by the PI. WEST is interfaced with QuantumEspresso (http://www.quantum-espresso.org) which is also open-source and already installed in the targeted HPC architecture. The code uses a hybrid OpenMP-MPI parallelization scheme and, although self-contained, can take advantage of highly optimized libraries like BLAS, LAPACK, SCALAPACK, FFTW3, ESSL, PESSL, GDLib and GSL. WEST implements a multiple level of MPI communicators providing massive parallelism even for large systems. West have demonstrated close to linear parallel scaling up to 3000 cores with fair parallel scaling to 6000 cores on the BG/Q Mira at ANL-ALCF for a liquid/solid interface model containing 492 atoms and 1560 valence electrons[2]. Justification of the number of requested core hours: In order to create the WEST GW database of results for a set of 100+ molecules we need a total of 1,250,000 h. In order to test the implementation of HDF5 with WEST on large systems of unprecedented size such as nanostructures and molecular crystals, we need to run transferability tests and production runs that will require approximately 500000 h. In order to test the the new pre- and post-processing set of productivity tools in WEST we need approximately 250000 h. The total number of hours requested is 2 million CPU hours. Team qualification: Marco Govoni is a research associate at the Institute for Molecular Engineering in the Materials Science Division of Argonne National Laboratory. He also has a joint visiting scientist appointment at the University of Chicago. He holds a PhD in Nanoscience and Nanotechnology from the University of Modena and Reggio Emilia, Italy. He was postdoctoral scholar at the University of California Davis (UC Davis). His main interest is modeling and understanding materials for energy research. He is the team coordinator and director of software engineering of the massively parallel many-body perturbation code WEST. He has collaborated with the Supercomputing Application and Innovation Department (SCAI) at CINECA, Italy in the past where in 2012 he was awarded an ISCRA scholarship for petascale software development. His experience in high performance computation and his experience as a software developer guarantees an efficient, productive and solid management of the allocated resources . The quality of the applicant’s HPC-related work is supported by publications in high-level journals, e.g. Nature Photonics and Journal of the American Chemical Society [12-13]. He will provide an optimal management of the resources allocated for MBPT simulations. His source of funding comes from projects led by Prof. G. Galli at ANL and UChicago. He will be heading the simulations and the scientific discussion with the other team member. Jonathan H. Skone is a staff member of the Research Computing Center at the University of Chicago, where his primary responsibility is to provide HPC solutions to the Institute for Molecular Engineering. He previously held a postdoctoral appointment within the Materials Science Division at Argonne National Laboratory where he pursued projects related to water catalysis with oxides and methodological development for condensed matter systems. He holds a PhD in Chemistry from Pennsylvania State University. His experience, from past projects carried out at as a postdoc at both Brookhaven National Laboratory and UC Davis, include the modeling of materials and molecules for electrocatalysis and photocatalysis. With the PI, he authored publications in hybrid functionals development [7-9], whose applicability to large scale modeling of aqueous electrolytes will be explored in the recently awarded ALCC project. The inclusion of Jonathan in the team is motivated by his expertise on sim ulations with the WEST code. He will serve as the point of contact for maintaining the database of benchmarked systems and will develop the necessary routines in order to interface WEST with the targeted I/O libraries (HDF5 and NetCDF). Being a research consultant for the Institute for Molecular Engineering, he will make sure that the project is well embedded in the newly established engineering research center, whose mission is to build a partnership between the University of Chicago and ANL which will be bolstered by this increased interaction and scientific collaboration. References: [1] G. Onida, L. Reining, and Angel Rubio, Review of Modern Physics 74, 601 (2002). [2] M. Govoni, and G. Galli, J. Chem. Theory Comput. 11, 2680 (2015). [3] Y. Ping, D. Rocca, and G. Galli, Chem. Soc. Rev. 42, 2437 (2013). [4] https://qboxcode.org [5] P. Giannozzi et al., J. Phys.: Condens. Matter 21, 395502 (2009). [6] http://th.fhi-berlin.mpg.de/site/uploads/Publications/GW100-1_20150518.pdf [7] J. H. Skone, M. Govoni, and G. Galli, Phys. Rev. B 89, (2014). [8] J. H. Skone, M. Govoni, and G. Galli, “Using Dielectric Properties to Design Nonempirical Hybrid Functionals for Accurate Electronic Structure of Molecules and Materials.” Phys. Rev. B To be submitted [9] A. Gaiduk, M. Govoni, J. H. Skone, and G. Galli, “The Photoelectron Spectrum and absolute band positions of ions in water using many-body perturbation theory with various DFT reference starting points.” To be submitted [10] H. Seo, M. Govoni, and G. Galli, “Design of defect spins in piezoelectric aluminum nitride for solid-state hybrid quantum technologies.” Submitted [11] https://www.hdfgroup.org/HDF5/ [12] M. Govoni, I. Marri, and S. Ossicini, Nature Photonics 6, 672 (2012). [13] I. Marri, M. Govoni, and S. Ossicini, J. Am. Chem. Soc. 136, 13257 (2014). Industry partnership: Project URL: http://www.west-code.org/ Requested allocation: 50000 Q1: 0 Q2: 0 Q3: 0 Q4: 50000 Justification: See project description under software readiness. Storage requirements: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: Meso_Nuclear
by accounts＠lcrc.anl.gov 15 Sep '15

15 Sep '15

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Linyun Liang Applicant's institution: ANL Applicant's division: MCS Project Name: Meso_Nuclear Project title: Gas bubble kinetics in an irradiated U-Mo using a mesoscale simulation approach Associated funding: Nuclear energy system Other Systems: Science: Understanding and predicting the irradiation induced structural damage in nuclear materials is of great interest to the nuclear community for developing new fuels for nuclear fission reactors. U-Mo alloys have been developed as a low enrichment metallic fuel under the Reduced Enrichment for Research and Test Reactor (RERTR) Program. They displays a predictable and stable swelling by fission products at moderate burnups and typical operation temperatures. Fission gas atoms Xe and Kr are generated in a nuclear fuel at the rate of about 0.25-0.30 atoms per fission as a result of decay of the primary fission products. About 7 times more Xe is produced than Kr. These gas atoms are very insoluble in the fuel that they do not react chemically with any other species. The release of fission gas product from the fuel has the primary effect of increasing the inner pressure of the fuel pin, and consequently, is considered as one limiting factor for the discharge burnup of a fuel rod. Project description: A mesoscale simulation approach is developed to study the gas bubble evolution kinetics in irradiated U-Mo alloy fuels. Density functional theory is first utilized to predict the defect formation energies, the Xe-U-Mo interfacial energy, and the elastic constants of U-Mo alloys, among other quantities. These obtained material properties are then incorporated into a mesoscale model to study the kinetics of gas bubbles in U-Mo under irradiation. By developing a new free energy model of Xe gas bubble based on the van der Walls equation of state, the gas bubble pressure can be estimated. The effects of vacancy and self-interstitial atom (SIA) diffusion on the gas bubble pressure will be further investigated and validated. The events employed in the simulation are the absorption and emission of a vacancy, a SIA, and an interstitial U by a single Xe bubble, while an event associated with SIA loops or substitutional U is neglected here for simplicity. We wi ll systematically investigate the effect of Xe, vacancy, and SIA concentration, fission defect generation and their recombination, sink strength, grain boundaries, and elastic interaction on the growth kinetics of gas bubble. The swelling of U-Mo due to fission gas bubble will be simulated and compared to experimental observations. Our study aims to provide insight into understanding of the structural damage in irradiated U-Mo materials. In our phase field model, we have four highly non-linear, coupled partial differential equations which must be solved numerically. In the meaning time, the elastic equilibrium equation is also included in each time step, which requires to obtain the elastic solution at the equilibrium state for each time step and can significantly increase the cost of the simulation time. And also in order to avoid the partial loss of interfacial energy, we will extend our model to three dimensional system. Therefore, to effectively solve this problem, we will need about 150,000 core hours including the 6,000 core-hour to debug and test the algorithm and code and 144,000 core-hours (4 cores/node * 4 nodes * 10 hours of run time * 800 different runs) for the nucleation and growth of gas bubbles calculations. We will use our developed FFTW code to solve the problem. By using the Fusion and Blues computing resources will greatly accelerated the code development and calculations. In conclusion, the total core-hours requested for this project throughout the FY2016 will be 150,000 core-hours. Fig. 1 Simulated results of gas bubbles nucleation and growth under the irradiated U-7Mo. Industry partnership: Project URL: Requested allocation: 150000 Q1: 0 Q2: 0 Q3: 0 Q4: 150000 Justification: Storage requirements: 1TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for radix
by accounts＠lcrc.anl.gov 15 Sep '15

15 Sep '15

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Pavan Balaji Project Name: radix Division: MCS Project title: Scalable Parallel System Software Associated funding: DoE core, SciDAC, NSF, FASTOS Other Systems: breadboard (unlimited) ALCF Mira/Cetus/Tukey (50,000,000 cpu hours) Science: Our mission is to develop the technologies required to dramatically increase the productivity of scientists developing applications for parallel supercomputers. The focus is fourfold: integration of parallel programming tools, reuse of parallel program components, development of scientific computing toolkits and portable libraries, and exploration of requirements of future parallel computers. Project description: This allocation will be used for optimizations to various systems software components: (1) programming models and runtime systems, (2) data I/O and file systems, (3) fault tolerance, (4) operating systems, and (5) data analysis and visualization. Primary goal of the project is to perform research on improving the core systems software on high-end computing platforms. The radix allocation covers a large number of cross-cutting projects involving subsets of researchers in the group. Three core projects are presented here, but some of the techniques used are shared between them and are not easily separable (e.g., MPI improvements or resilience improvements): 1. Fault Tolerance Techniques: - Local memory management * SDC Detection through Data Monitoring + Detect outliers on application data through one-step ahead prediction * Energy Efficient Multilevel Checkpointing + Power monitoring integration into resilience libraries and energy-aware ckpt. scheduling * I/O Prediction for Multilevel Checkpoint + Inter-application coordination and I/O collision avoidance - Distributed memory management * MPI ULFM + Improvements to MPICH to deal with faults * Resilient-MPI and FTI Integration + Experimentation and evaluation of FTI coupled with resilient-MPI 2. Low-level Hybrid Programming with MPI and threads: - Communicating with Massive Threads * Integrate threads with MPI to support communicating with light-weight threads - Recently funded project (Argo) that deals with improvements to both MPI and the threading model * MPICH improvements to play well with user-level threads * OS improvements to provide light-weight threads that "understand" MPI concepts 3. Data I/O, RMA and Active Messages: - Large-Scale Storage System Prototype and Simulation * Evaluate future storage system using discrete event simulation with MPI and InfiniBand - Dynamic Locality-Aware Workflows * Dynamically distribute the tasks among nodes using Asynchronous Dynamic Load Balancer - Parallel I/O autotuning * Automate the selection of good configuration parameters for HPF5, MPI-IO and the parallel file system Industry partnership: Project URL: http://www.mcs.anl.gov/group/pmrs/ Current FY Hours Used: undetermined amount New FY Requested allocation: 2000000 Q1: 500000 Q2: 500000 Q3: 500000 Q4: 500000 Justification: 2M core hours requested: Three core projects. All projects have been demonstrated at scale in the previous years and form the core of the system software stack used in production on the Fusion and Blues machines. We expect to run around 75 studies for each project on Blues and Fusion, scaling from 1 node to 256 nodes (average runtime of 30 minutes) - Blues: 3 (projects) x 75 (studies) x 512 (1 to 256 nodes) x ½ (hour) x 16 (cores) = 920K - Fusion: 3 (projects) x 75 (studies) x 512 (1 to 256 nodes) x ½ (hour) x 8 (cores) = 460K Some smaller projects: We expect to run around 75 studies for all projects, scaling from 1 node to 256 nodes (average runtime of 30 minutes) - Blues: 75 (studies) x 512 (1 to 256 nodes) x ½ (hour) x 16 (cores) = 310K - Fusion: 75 (studies) x 512 (1 to 256 nodes) x ½ (hour) x 8 (cores) = 155K A small fraction of this would be used for debugging. Storage requirements: No large storage requirements, since the data generated will not be persistent. Thank You, The LCRC Accounts System

1 0