Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Kendra Letchworth-Weaver Applicant's institution: ANL Applicant's division: NST Project Name: JDFT-Surfaces Project title: Joint Density-Functional Theory Investigations of Realistic Surface Structure Associated funding: LDRD/Named Fellow Other Systems: Carbon at Center for Nanoscale Materials (510,000 hours) Science: The surface structure of a surface or nanoparticle in contact with liquid is often different from the structure in vacuum. This structure may be probed experimentally by techniques such as X-ray reflectivity or pair distribution functions, however the results often require interpretation and validation. Using Joint-Density-Functional Theory, which enables quantum-mechanical calculations of a solute system (such as an surface) in contact with a liquid environment, we will make predictions of surface structure in liquid which will provide unique insights into related experimental studies. By avoiding the statistical sampling over configurations of the liquid required by classical and ab initio MD, JDFT enables rapid screening of surface terminations in liquid environment for agreement with experimental data. In addition, JDFT naturally provides surface energies in solution and as a function of voltage, allowing us to synthesize experimental and theo retical information to determine surface structure under realistic electrochemical and growth conditions (as in doi:10.1021/jacs.6b03338). Project description: We plan to consider several systems within this project: 1. The surface structure of geological materials in water, such as alumina or hematite, for direct comparison to X-ray reflectivity data from the group of Paul Fenter in CSE. We will perform AIMD to sample the surface alone while treating the liquid through JDFT. For alumina and hematite each, we will consider roughly 5 terminations in AIMD, requiring 10 ps of statistics for each. Parallelized over 9 NVidia C2075 GPUs, 1 ps of statistics for these surfaces requires roughly 24 hours of walltime. We expect that parallelizing over 6 faster Tesla K40 GPUs will yield the same walltime (2 materials x 5 terminations x 10 ps x 24 hours x 6 GPUs x 32 core hours=460800 hours) 2. The surface structure of SrTiO3 during growth of Ruddlesden Popper stacking faults, for comparison to resonant X-ray reflectivity data from the group of Dillon Fong in MSD. Though there is no liquid present, these measurements were taken at high temperatures so AIMD is also required to account for thermal motion of the surface. The number of calculations required is similar to the values for the geological materials above. (1 material x 5 terminations x 10 ps x 24 hours x 6 GPUs x 32 core hours= 230400 hours) 3. How the shape of IrO2 nanoclusters for solar water-splitting applications changes in the presence of liquid (in collaboration with Theory and Modeling group at CNM). We will also consider how the binding energy of oxygen at different adsorption sites on the nanoparticle changes when liquid is included. The initial low-energy IrO2 nanoparticle shapes will be determined by a genetic algorithm search using force fields to represent the atomic interactions. We will then consider approximately 10 low energy nanoclusters in liquid using JDFT. Finally, we will study approximately 10 oxygen binding sites on the lowest-energy nanocluster in solution. These calculations are not currently MPI parallelizable (though they are OpenMP and GPU parallelized to run on a single node and MPI parallelization may be implemented before the end of the allocation). These geometry optimization calculations will each take 1 week of walltime to run in vacuum and another week in liquid. (2 solvation states x 20 calculations x 7 days x 24 hours x 32 core hours=215040 hours) The total production run hours above add to 906,240 hours. We estimate using roughly an additional 100,000 hours (10% of the requested allocation) for debugging and convergence tests. We will use the electronic structure code JDFTx, an open-source plane-wave DFT code co-developed by the PI and written in C++. It is available for download at http://jdftx.org. We require the following packages to be installed: • cmake (>=2.8) • g++ (>=4.6) • libgsl0-dev • libopenmpi-dev (or equivalent for other MPI distributions) • libfftw3-dev • libatlas-base-dev (provides blas and cblas) • liblapack-dev (alternatively, math operations using Intel’s MKL are possible) See http://jdftx.org/Compiling.html) for details. I will be working in collaboration with Maria Chan as well as Fatih Sen on this project for 3 team members in total. Industry partnership: Project URL: http://jdftx.org Requested allocation: 1000000 Q1: 250000 Q2: 250000 Q3: 250000 Q4: 250000 Justification: JDFTx can be run in either a hybrid GPU-MPI mode for smaller memory jobs or an OpenMP-MPI mode for larger memory jobs, offering excellent performance and scaling. GPU parallelization offers a speedup of 3-5x over a single CPU. The JDFT liquid experiences near-linear scaling when MPI-parallelized over orientations (typically 24-144 orientations are required) The electronic DFT experiences near-linear scaling when MPI-parallelized over k-points. Coulomb truncation algorithms implemented in the code also reduce the size of the simulation grid for JDFTx wavefunctions. As an example, scaling data for a small Al2O3 surface supercell (34 atoms, 9 k points) on NERSC Edison is provided below in vacuum. For the vacuum calculations, we perform initialization and a single electronic wavefunction minimization. Each MPI process was run with 12 OpenMP threads (2 processes per 24-core node). MPI Processes time 1 449 s 2 249 s 3 165 s 5 114 s 9 71 s The large allocation is requested because GPU calculations are charged double compared to CPU calculations, yet we will be able to utilize MPI parallelization over GPU’s. Alternatively, if the GPU is not available, we could MPI parallelize over CPU’s, though we would require the additional factor of 3-5 in walltime. Storage requirements: 1 TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System