Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: John J. Low Applicant's institution: ANL Applicant's division: MCS Project Name: WEST Project title: GW for the materials science community: extending transferability, benchmarking and addition of new productivity tools within the WEST code Associated funding: LDRD at ANL-MSD (PI), co-PI is staff at the UChicago’s Research Computing Center (RCC). MICCoM (DOE-funded computational center directed by G. Galli) Other Systems: Midway (RCC at UChicago) Science: The investigation of new materials that can extract, store and use energy in a more efficient way represents a key challenge of basic energy science research. First principle simulations based on Density Functional Theory (DFT) and Many-Body Perturbation Theory (MBPT) are expected to accelerate materials discovery while substantially lowering manufacturing costs. Recently we have released the open-source code WEST (www.west-code.org), which implements the DFT+MBPT scheme adopting several algorithmic and numerical advantages that are specifically targeted to tackle large scale simulations, proving a good scalability on >200k cores. In this project we propose to broaden the WEST user base and the number of users capable of using leadership computing resources by: i) establishing a publicly accessible database of results, ii) improving the transferability of data files across multiple platforms through the development of an interface with portable I/O l ibraries (e.g. HDF5 and NetCDF), and iii) developing pre- and post-processing tools to improve productivity with WEST. All codes used in this project and developed by the team are open-source, making them freely available to the scientific community of users. The findings of this project will be published in peer-reviewed journals, and the results of the simulations will be exported in the form of digital contents to be disseminated at national and international conferences, and in press articles and webpages. The search for new materials that can extract, convert and store energy with greater efficiency represents a grand challenge of basic energy sciences, with federal funding agencies providing continued support of this mission at the fundamental research level. In the field of materials science, researchers are striving to find non-toxic, disposable, cheap and durable materials that can be made more functional than those in current use. In particular, with the Materials Genome Initiative strategy (www.whitehouse.gov/mgi), first principle simulations based on density functional theory (DFT) are expected to accelerate discovery while substantially lowering manufacturing costs. Important quantities in the design of applications for photovoltaics, photocatalysis, and quantum information systems are computable physical properties like, band gaps, band edge positions, defect energy levels and absorption spectra. These properties are often poorly described by semilocal DFT. For more r eliably accurate physical descriptors it is typically required to compute these properties by combining DFT with Many-Body Perturbation Theory (MBPT), which requires the solution of the quasiparticle equation [1]. Along with an increased accuracy comes a higher computational cost, which has been the main limitation of these techniques to large-scale applications thus far. Project description: John Low is acting as temporary PI until Marco Govoni's 593 has been completely processed. This allowed this project request to be submitted for a timely review by the allocation committee. Eventually John Low will most likely be removed as PI after Marco gets an LCRC account and becomes the formal PI for this project. Jonathon Skone will be a co-PI. There will be only two members of this project. Recently the PI(Marco Govoni) has released the open source code WEST (www.west-code.org), which implements the DFT+MBPT scheme adopting several algorithmic and numerical advantages that are specifically targeted to tackle large scale simulations. Details about the code can be found in Ref. [2], where scalability on >200k cores is discussed and systems of unprecedented size are simulated. In WEST the Dyson equation is solved in an approximate manner using iterative techniques that make use of matrix-free operations and density-based linear response theories [3]. A perturbative solution of the Dyson equation is obtained in two steps, the first of which involves an iterative diagonalization of the dielectric matrix via a Davidson diagonalization. At each step several linear systems are solved using the preconditioned conjugate gradient method to obtain the linear variation of the charge density, in the framework of Density Functional Perturbation Theory (DFPT). The second step i nvolves an expansion of the Green’s function and the screened Coulomb potential in a separable manner, followed by a frequency convolution via a Lanczos iterative algorithm. The iterative diagonalization of the dielectric matrix uses a deflation technique that does not require the calculation of already converged eigenvectors. Overall, the combination of recasting the solution of quasiparticle equations using mathematical techniques with the exploitation of massively parallel algorithmic implementations allows WEST to provide highly accurate physical descriptors for systems of unprecedented size (1000s of electrons or more). WEST uses as input the electronic structure evaluated with Density Functional Theory (DFT) codes such as Qbox [4] and Quantum Espresso [5], which solve the Kohn-Sham equations using the plane wave pseudopotential method. Code interoperability strongly calls for the adoption of a transferable I/O paradigm, which favors the creation of an easily convertible and architecture independent database. By interfacing WEST with parallel and transferable I/O libraries, eg. HDF or netCDF, we will solidify the use of a multi-platform workflow paradigm for production runs. Data transferability will also facilitate the ability to post-process data, obtained on remote resources, on local machines with tools under development. Finally, in order to provide a reference database of GW simulations, we propose to create a benchmark of WEST results, that will be published in a refereed journal article and made publicly accessible through the WEST website.

From the resources allocated within this project, we will broaden the WEST user base and the number of users capable of using leadership computing resources.

This project supports time allocation on Argonne’s Laboratory Computing Research Center Blues cluster. The activity of this proposal will be coordinated with the materials simulation environment of MICCoM, the newly establish computational DOE center (ANL-PI G. Galli), and eventually be integrated into it. The project will also be interfaced with the research activity of the Galli group at the Institute for Molecular Engineering (http://galligroup.uchicago.edu). In the following we outline and describe in detail the three aims of this proposal. Aim #1: Establishing a publicly accessible database of GW results procured using WEST to serve as both a reference and a benchmark for the scientific community. The techniques unique to WEST, whereby the arduous task of converging the unoccupied manifold is eliminated, provide an improved state-of-the-art methodology for MBPT electronic structure calculations. These results may serve as a benchmark to other traditional MBPT codes where the accuracy is strongly dependent on the convergence of the unoccupied states and often times is approximated or extrapolated. We intend to build our database of WEST results starting from a recently proposed data set of 100 molecules (GW100) [6]. The previously published quasiparticle results compared several traditional GW codes, but there were several drawbacks in this reference data set for which we will improve upon by building a benchmark database of WEST results. The first issue with the aforementioned reference GW data set is that all results reported are for traditional GW methodologies that require the convergence of the unoccupied states. With WEST results the need for converging the unoccupied manifold of states is removed thus providing a superior reference point that even the traditional GW codes could take as a benchmark for establishing their convergence. Another issue lies with the DFT starting point reference to compute the QP corrections with a single step GW approach. Since the wavefunctions and the dielectric function (screened Coulomb interaction W) are not iteratively solved till self-consistency in a single step GW approach, the starting point upon which perturbative corrections are evaluated is of critical importance. The importance of the starting point, especially for systems where the dielectric screening and wavefunctions are markedly different depending on the starting point reference DFT is discussed in recent papers where WEST was used [7-10]. In the database of WEST results we aim at establishing GW results obtained with hybrid density functional reference starting points. We also intend to expand the set of reference data for 100 molecules to include a dozen or more molecules of relevance to the organic semiconducting industry, namely the set of variously branched polycyclic aromatic molecules (e.g. rubrene and pentacene) and charge transfer complexes such as tetracyanoquinodimethane--tetrathiafulvalene, for which presently no established reference GW results are available. The results of the benchmark will be published and serve as reference for contemporary and future GW calculations. All input and output files (inclusive of convergence tests) will be made available on a dedicated section of the WEST website. The large volume of meta-data will be stored at the Research Computing Center at the University of Chicago and served up through the WEST site. The data will provide a benchmark for future theoretical implementations and be inclusive in the set of reference test calculations used to ensure the fidelity of the WEST source code. The generated database will be used also by the WEST integrity software test suite known as the WEST Regression Test (WRT), which is comprised of a series of input tests, that are run on Midway at the RCC on a nightly basis checking the output generated from each test input file against the repository of verified correct output. The set of 100+ molecule quasiparticle computations with WEST requires approximately 1.25M hours of computing time. Aim #2: Improving transferability of data files generated and used by the massively parallel software WEST. We intend to further enhance the transferability of data files generated from parallel I/O in WEST through the implementation of HDF5 [11]. This aim requires the use of a set of diverse high performance computing architectures in order to test the transferability of data files generated from parallel I/O and will be addressed by initiating tasks either at ALCF or LCRC, then moving the initial data files to be restarted either at the LCRC or the RCC at the University of Chicago. We are requesting a total of 500k core hours of time at LCRC for the purpose of testing the transferability of data files generated by WEST at LCRC’s Blues cluster and transferred within Argonne to the ALCF’s Mira supercomputer or offsite to the Research Computing Center (RCC) cluster at the University of Chicago. In addition to making data transferability possible across diverse computer cluster architectures, this proposal serves to bolster the communication and collaborative efforts between the computational centers at Argonne National Laboratory (LCRC) and the University of Chicago (RCC). There are several steps involved in the procurement of results using WEST. Prior to even running WEST, the user needs to obtain a structurally converged geometry and a ground state electronic structure which may be obtained with other scientific software such as LAMMPS, Qbox, or Quantum Espresso etc. This step, due to lower degree of complexity and computational cost may be run on a mid-sized cluster such as (LCRC or RCC at the University of Chicago). The data files generated for the ground state electronic structure would then need to be transferable if the user is to then use this information to initiate a WEST calculation on a larger sized supercomputer such as the ALCF’s Mira (or the new upcoming ALCF machines Theta and Aurora). It is also often the case that a resource allocation may expire or become depleted and the user must move their incomplete WEST job to another resource center. In order for the job to restart from these data files generated on a different archit ecture a transferable I/O data structure is needed. Furthermore, once results are obtained using WEST, often times it is necessary to run postprocessing tools to further analyze the data and/or make meaningful plots. This is a much less computationally demanding task that the user can achieve on a modest desktop or laptop computer, but in order to do so the data files need to be generated from transferable parallel I/O to be further manipulated. This aim will also result in a technical paper describing the parallel I/O structure that will be published in a peer-reviewed journal. We also expect this project to bolster the relationship between ANL-LCRC/ALCF and the University of Chicago Research Computing Center. Aim #3: Pre- and post-processing tools to improve productivity with WEST. The creation and testing of new pre- and post-processing tools will provide increased user productivity and extend the reach of systems suitable for analysis with WEST. Of the existing post-processing tools (e.g. Wannier center analysis, Spectral function plot, STM imaging), improvement in the parallelization of these routines will provide for improved efficiency of resource use and a quicker turnaround time in the data analysis of WEST results. The new post-processing tools we intend to implement are those which will have an immediate impact on the interpretation and data visualization of WEST results. These new post-processing tools include a set of python modules to graphically plot the quasiparticle corrected photoelectron spectra, the associated lifetimes, and visualization plots of the dielectric function. To expand the scope of applicable systems that can be treated with WEST we will implement the routines necessary to handle more general lattice systems, making it possible to then handle, for instance, monoclinic crystal systems--typical of many semiconducting organic molecular crystals. To carry out the needed testing of the implementation of arbitrary unit cell input, the new post-processing tools for plotting quasiparticle photoelectron spectra / lifetimes, and the parallelization of the existing post-processing tools, an additional 250k hours of computing time is required. Software Readiness: We report in the following, the details of the WEST code, which will be used extensively in this project. WEST (Without Empty STates). Accurate electronic structures will be computed using MBPT. Quasiparticle corrections will be computed using the G0W0 approach where the electronic self-energy is solved using iterative techniques developed by the PI [2]. The details of these techniques, that do not require the explicit calculation of empty electronic states and fit the inversion of large dielectric matrices (typical problems of large scale systems), are given in a recent review paper [3]. The screened Coulomb interaction W is obtained in the random phase approximation by representing the dielectric screening with a projective eigendecomposition basis set. The iterative diagonalization of the screening with the Davidson’s algorithm is started with random potentials and then converged in 4-6 steps. At every iteration, each eigenpotential gives rise to a stand-alone Sternheimer problem where the perturbed density is computed. Because each perturbation can be treated independently, this part of the algorithm i s embarrassingly parallel. GW self-energies are then obtained using a combined Lanczos-Deformation Contour approach, analogously parallelized. WEST is an open-source software (www.west-code.org) which is maintained by the PI. WEST is interfaced with QuantumEspresso (http://www.quantum-espresso.org) which is also open-source and already installed in the targeted HPC architecture. The code uses a hybrid OpenMP-MPI parallelization scheme and, although self-contained, can take advantage of highly optimized libraries like BLAS, LAPACK, SCALAPACK, FFTW3, ESSL, PESSL, GDLib and GSL. WEST implements a multiple level of MPI communicators providing massive parallelism even for large systems. West have demonstrated close to linear parallel scaling up to 3000 cores with fair parallel scaling to 6000 cores on the BG/Q Mira at ANL-ALCF for a liquid/solid interface model containing 492 atoms and 1560 valence electrons[2]. Justification of the number of requested core hours: In order to create the WEST GW database of results for a set of 100+ molecules we need a total of 1,250,000 h. In order to test the implementation of HDF5 with WEST on large systems of unprecedented size such as nanostructures and molecular crystals, we need to run transferability tests and production runs that will require approximately 500000 h. In order to test the the new pre- and post-processing set of productivity tools in WEST we need approximately 250000 h. The total number of hours requested is 2 million CPU hours. Team qualification: Marco Govoni is a research associate at the Institute for Molecular Engineering in the Materials Science Division of Argonne National Laboratory. He also has a joint visiting scientist appointment at the University of Chicago. He holds a PhD in Nanoscience and Nanotechnology from the University of Modena and Reggio Emilia, Italy. He was postdoctoral scholar at the University of California Davis (UC Davis). His main interest is modeling and understanding materials for energy research. He is the team coordinator and director of software engineering of the massively parallel many-body perturbation code WEST. He has collaborated with the Supercomputing Application and Innovation Department (SCAI) at CINECA, Italy in the past where in 2012 he was awarded an ISCRA scholarship for petascale software development. His experience in high performance computation and his experience as a software developer guarantees an efficient, productive and solid management of the allocated resources . The quality of the applicant’s HPC-related work is supported by publications in high-level journals, e.g. Nature Photonics and Journal of the American Chemical Society [12-13]. He will provide an optimal management of the resources allocated for MBPT simulations. His source of funding comes from projects led by Prof. G. Galli at ANL and UChicago. He will be heading the simulations and the scientific discussion with the other team member. Jonathan H. Skone is a staff member of the Research Computing Center at the University of Chicago, where his primary responsibility is to provide HPC solutions to the Institute for Molecular Engineering. He previously held a postdoctoral appointment within the Materials Science Division at Argonne National Laboratory where he pursued projects related to water catalysis with oxides and methodological development for condensed matter systems. He holds a PhD in Chemistry from Pennsylvania State University. His experience, from past projects carried out at as a postdoc at both Brookhaven National Laboratory and UC Davis, include the modeling of materials and molecules for electrocatalysis and photocatalysis. With the PI, he authored publications in hybrid functionals development [7-9], whose applicability to large scale modeling of aqueous electrolytes will be explored in the recently awarded ALCC project. The inclusion of Jonathan in the team is motivated by his expertise on sim ulations with the WEST code. He will serve as the point of contact for maintaining the database of benchmarked systems and will develop the necessary routines in order to interface WEST with the targeted I/O libraries (HDF5 and NetCDF). Being a research consultant for the Institute for Molecular Engineering, he will make sure that the project is well embedded in the newly established engineering research center, whose mission is to build a partnership between the University of Chicago and ANL which will be bolstered by this increased interaction and scientific collaboration. References: [1] G. Onida, L. Reining, and Angel Rubio, Review of Modern Physics 74, 601 (2002). [2] M. Govoni, and G. Galli, J. Chem. Theory Comput. 11, 2680 (2015). [3] Y. Ping, D. Rocca, and G. Galli, Chem. Soc. Rev. 42, 2437 (2013). [4] https://qboxcode.org [5] P. Giannozzi et al., J. Phys.: Condens. Matter 21, 395502 (2009). [6] http://th.fhi-berlin.mpg.de/site/uploads/Publications/GW100-1_20150518.pdf [7] J. H. Skone, M. Govoni, and G. Galli, Phys. Rev. B 89, (2014). [8] J. H. Skone, M. Govoni, and G. Galli, “Using Dielectric Properties to Design Nonempirical Hybrid Functionals for Accurate Electronic Structure of Molecules and Materials.” Phys. Rev. B To be submitted [9] A. Gaiduk, M. Govoni, J. H. Skone, and G. Galli, “The Photoelectron Spectrum and absolute band positions of ions in water using many-body perturbation theory with various DFT reference starting points.” To be submitted [10] H. Seo, M. Govoni, and G. Galli, “Design of defect spins in piezoelectric aluminum nitride for solid-state hybrid quantum technologies.” Submitted [11] https://www.hdfgroup.org/HDF5/ [12] M. Govoni, I. Marri, and S. Ossicini, Nature Photonics 6, 672 (2012). [13] I. Marri, M. Govoni, and S. Ossicini, J. Am. Chem. Soc. 136, 13257 (2014). Industry partnership: Project URL: http://www.west-code.org/ Requested allocation: 50000 Q1: 0 Q2: 0 Q3: 0 Q4: 50000 Justification: See project description under software readiness. Storage requirements: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System