September 2017 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Yearly Allocation Request for wrfchem_for_aerprof
by accounts＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Yan Feng Project Name: wrfchem_for_aerprof Division: EVS Project title: Investigation of Cloud-Aerosol Interactions Associated funding: NASA Earth Science Division Other Systems: NERSC Science: Marine low clouds have been recognized as a key component of the Earth’s energy budget due to their strong albedo and cooling effects. The radiative properties of these marine low clouds can be substantially altered by atmospheric aerosols, through changes in the cloud droplet number concentration and cloud optical thickness. Although these aerosol-mediated indirect effects have been widely observed and represented in models, their magnitudes constitute one of the largest uncertainties in our current understanding of radiative forcing. The goal of the project is to investigate aerosol-cloud interactions and the aerosol indirect effect in marine low clouds using NASA satellite datasets. High-resolution regional climate model WRF simulations will be conducted to provide the physical interpretation of the satellite-observed correlations and evaluate the relative importance of free tropospheric and boundary layer aerosols. Particular emphasis will be given to the aero sol vertical variability, aerosol humidification, and entrainment rate. Project description: In FY17, we have completed the model configuration and several test runs of the latest version of WRF model (3.9.1) and its precursor model WPS (3.9.1) on Blues. The latest WRF model is optimized and uses domain decomposition to divide total work load on distributed-memory systems. We have test-run WRF-Chem at three resolutions: 30km, 12km, and 4km for a 30°×40° domain (~20 50°N; ~152°W–112°W). Total grid number in the 4-km configuration is up to 19 millions, currently scaled to 36 nodes. The initial results are being processed and evaluated before longer simulations. In FY18, we plan to perform 5-10 ensemble simulations with the new WRF model at 12 km and 4 km resolutions each. Those simulation experiments will be three-month for May-June-July of 2013. This is the season when marine low clouds off the California coast are most abundant. We will scale the WRF model to more node numbers, especially for 4 km resolution runs. In addition, we will work on testing the chemistry module with WRF. Chemistry initial and boundary conditions will be obtained from the MOZART-4 global chemical transport model outputs. The default anthropogenic emissions in the U.S. are based on the 4-km the Environmental Protection Agency (EPA) 1999 National Emissions Inventory (NEI) dataset in WRF-Chem and will be updated for sulfate, organic carbon (OC), and black carbon (BC) from the NEI 2011 inventory. Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 90000 Q1: 25000 Q2: 20000 Q3: 25000 Q4: 20000 Justification: Storage requirements: 1 TB is suffient Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for LiSx
by support＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=LiSx

1 0

[LCRC Accounts] Yearly Allocation Request for perfect
by accounts＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: John J. Low Project Name: perfect Division: EGS, CLS, LCF, APS, MSD Project title: The Perfect Thermodynamics of Imperfect Materials Associated funding: LDRD Other Systems: ALCF/INCITE Science: Abstract: The goal of the project is to significantly improve the understanding and prediction of thermodynamic stability/metastability of “imperfect” (e.g., highly defective, non-stoichiometric, or doped) oxide material phases, via innovative theory (i.e., embedded uncertainty), advanced experiments at APS, and “intelligent” software (i.e., able to learn and quickly solve new problems). We envision building the knowledge and capabilities that will allow, over the next decade, the prediction of thermodynamic properties of imperfect materials, with impact on materials design, synthesis, and smart manufacturing. Furthermore, we expect this methodology to accelerate the development of the materials genome and the next generation computers. Science: We focus on high-k dielectric materials for complementary metal-oxide-semiconductor (CMOS), which are of particular importance for creating Dynamic Memory Allocation (DRAM) devices. Many CMOS properties strongly depend on material defects such as vacancies, interstitials, defect clusters that occur during synthesis, and thermal treatment. The inclusion of other chemical elements (e.g., dopants) in CMOS can significantly change physical properties such as thermal conductivity, electrical conductivity, and magnetism. Far from representing a drawback, the understanding imperfections, including defects and dopant, leads to prediction and control of properties and the design of post-CMOS materials for the next generation of electronic devices. There is, however, a challenge: the thermodynamic stability of CMOS phases – a crucial element for any application – is hard to evaluate, especially at high temperatures, close to the phase transformations or the melting point. In this project, we develop the theoretical, experimental, and computational knowledge and tools for predicting the thermodynamic and kinetic stability/metastability of defected CMOS phases at temperatures ranging from 0K to melting. As a proof of principle, we will study the high k dielectric in complementary metal–oxide–semiconductor (CMOS), using as a prototype hafnia, HfO2. Hafnia has unusual electronic and magnetic properties that make it attractive to both science and technology. One of the most critical applications is based on hafnia’s high-k dielectric properties, which are of particular importance for creating Dynamic Memory Allocation (DRAM) devices. In 2007, Intel decided to replace silicon oxide with hafnia as a gate insulator for field-effect transistors. Current and future technology (sub 14nm) are using “hafnia based” materials (including additional elements). Many CMOS properties strongly depend on materials defects such as vacancies, interstitials, defect clusters (Figure 1), and how the material was synthesized and pretreated. We will extend this studies to defects in the tetragonal, cubic and molten HfO2. Throughout the project, we will compare the computational results to available experimental and theoretical data. We will focus on oxygen vacancies, hydrogen interstitials, and nitrogen dopants, which are common defects in HfO2 high k-dielectrics. Additional work will model the crystallization of amorphous hafnia. Project description: In a preliminary study, we have calculated enthalpies of several Hf-O phases as a function of temperature using ab initio molecular dynamics (AIMD). The calculations involved a few dozen trajectories spanning a temperature range from 0K to a 4000K (above the melting temperature of hafnia) for each model of the perfect and defective phases. Each trajectory involves 10 picoseconds of dynamics (10,000 time steps) or 14 hours on 16 Broadwell nodes (8064 core-hours). Calculating two dozen trajectories requires 193,536 core-hours. We have completed trajectories for perfect and defective monoclinic, tetragonal and cubic polymorphs during bebop’s early user period. We still need to complete the calculations for hydrogen interstitials, nitrogen interstitials, and the molten HfO2 phase. The results of this work will be used to develop a force-field Hafnia using Machine Learning by Y. Ghadar and A. Vazquez Mayagoitia. We estimate that the force-field develop ment work will require 2,000,000 core-hours to complete. Hafnia in semiconductors is a thin (10-100 nm) amorphous layer between layers of Si. The synthesis process determines the structure of this nanolayer. Typically, atomic layer deposition or chemical vapor deposition is used to fabricate these monolayers. We will follow the procedure proposed by Shirazi and Elliot [J. Comp. Chem. 35(2014)244] and use Kinetic Monte Carlo (KMC) to predict the structure of the layer of amorphous HfO2 grown on Si during ALD or CVD. The predicted reaction rates, needed as input in KMC, will be derived from density functional calculations. To solve the KMC equations, we will use Eon [Henkelman and Jonsson, JCP 115(2001)9 657] and SPPARKS [Plimpton et al. JCP 128(2008)205101]. The KMC simulations will require 250,000 core-hours. A critical parameter is the crystallization temperature of the amorphous layer of hafnia. Crystallization is detrimental to a dielectric property because it can lead to current leakage. To avoid leakage, the processing temperature of the DRAM chip must be less than the crystallization temperature. Since the amorphous phase is metastable, the standard process of estimating free energies from energies and phonon bands to predict phase transitions is not adequate. We propose to use Forward Flux Sampling (FFS) [Allen et al. J. Phys.: Condens. Matter. 21(2009)463102] to model the crystallization process, predict rates, and predict the crystallization temperature. We will use reactive force-fields in LAMMPS to model this process and will develop LAMMPS scripts to conduct FFS through collaboration with Art Voter (LANL). We estimate that this study will require 100,000 core-hours. All data resulting from theory and calculations will be evaluated using Bayesian Analysis (BA) to determine the optimal models (i.e., mathematical expressions) that relate the free energy of the defected oxide phases to temperature and defect concentration (first and second year). To this end, we will use a Non-Naive Bayesian Network machine learning algorithm and software to process that will analyze the training and validation data sets in an unsupervised mode. The data generated by the DFT and KMC computations described above will enrich the diversity of the samples and reduce the final error in our estimations. The computationally intensive calculations will improve the evaluation of the stability and metastability of phases and will add uncertainty intervals, using a methodology developed by the PI and co-authors (M. Stan and B. Reardon., CALPHAD, 27 (2003) 319). We estimate that 100,000 core-hours will be required for these calculations. We will explore the scalability of alternative electronic structure programs as part of this project. Since a majority of the CPU time consumed in this project is by electronic structure programs, we evaluated several DFT codes such as CP2K and VASP to select the most efficient and scalable electronic structure program. The evaluation shows that VASP6 with multiple k points is the most accurate for smaller unit cells and exhibits competitive performance. OpenAtom is a new high-performance electronic structure program which claims good scaling up to ten thousand cores on BG/Q and blue-waters. This scaling would improve throughput by a factor of ten, which would be extremely useful in AIMD studies. We plan to use this OpenAtom to model molten and amorphous HfO2. This study will require 1,000,000 core hours. Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 3500000 Q1: 875000 Q2: 875000 Q3: 875000 Q4: 875000 Justification: LAMMPS is a high-performance code which has been optimized to run on Broadwell, NVidia GPUs, and Knights Landing processors. Molecular dynamics for thousands of atoms will be used to model crystallization which will scale well to thousands or cores. See http://lammps.sandia.gov/bench.html for more details. Version 6 of VASP is a hybrid MPI/open executable which can scale to a thousand cores on the Broadwell processors on bebop and Sandy Bridges processors on blues for unit cells containing ~100 atoms of HfO2. We observe good scaling for up to 8 KNL nodes with performance competitive with the Broadwell nodes. VASP has been our main engine for generating thermodynamic data from first principles. See https://cug.org/proceedings/cug2017_proceedings/includes/files/pap134s2-fil… for details on the new version of VASP. OpenAtom is a high-performance electronic structure program which uses Charm++ and has an improved FFT algorithm. It claims good scaling up to ten thousand cores on BG/Q and Blue Waters. See http://charm.cs.illinois.edu/OpenAtom/research.shtml for details. Eon and SPPARKS are both designed to run parallel. Eon can run multiple LAMMPS jobs simultaneously while SPPARKS can run multiple KMC jobs and has demonstrated good parallel scaling to a thousand cores or more. See http://henkelmanlab.org/pubs/chill14_055002.pdf for details. Storage requirements: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: RSoXS
by accounts＠lcrc.anl.gov 07 Sep '17

07 Sep '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Wei Chen Applicant's institution: ANL Applicant's division: IME, MSD Project Name: RSoXS Project title: Quantitative Determination of Interfacial Profiles through Resonant Soft X-ray Scattering Associated funding: DOE-BES-MSE Other Systems: Science: Recently, interfacial properties in polymer materials have attracted abiding interests, given the richness in science and of technological importance. When two different polymeric materials form an interface, novel properties and functionalities emerge that are possessed by neither material separately, as the presence of interface influences generally all the thermodynamic parameters of a system. Therefore, understanding how polymers at the interface interact and control each other is highly desired and has applications spanning disciplines from medicine to geology, from climate science to nanotechnology. In this study, we propose to utilize resonant soft x-ray scattering (RSoXS) to quantitatively determine interfacial profiles in multiphase polymer thin films. In addition, the fast acquisition of RSoXS will enable in situ studies of the development of polymer interfaces. As can be anticipated, the simultaneous simulation and fitting of the large amo unt of RSoXS data, which were collected at Synchrotron source, remains challenging in terms of the power of single workstation computer. Therefore, we are proposing to run the analysis of RSoXS to get quantitative information on interfacial profiles on the clusters available at LCRC. Project description: RSoXS measurements are performed at the photon energies from 270 eV to 290 eV, from 380 eV to 410 eV, from 515 eV to 545 eV, and 680 eV to 720 eV with a step of 0.1 eV for an accurate reconstruction and determination of interfacial profiles. So each data set includes hundreds of the scattering data profiles. This project aims to simultaneously analyze these RSoXS data by modeling asymmetric interfacial density distributions for the parametrization of any arbitrary density profiles in the "effective-density model" through generalized skew symmetric probability density functions. A continuous density profile is generated and discretized into many independent slices of very thin thickness with constant density values and sharp interfaces. We are planning to use Matlab scripts that we have developed and functions implemented in Curve Fitting, Optimization, Global Optimization, and Parallel Computing toolboxes in Matlab on Blues/Bebop. For a 12-core Mac P ro, the simulation and fitting take about 15-20 days (4000-6000 core-hours) to yield a reasonable model. If the genetic algorithm is used, it takes an even longer time. So, the calculations will require for 31000 core-hours per quarter for 5 different runs plus 1000 development hours. In total, these simulations and fittings will require 124,000 core-hours. There will be two project members, including myself. Industry partnership: None Project URL: Requested allocation: 124000 Q1: 31000 Q2: 31000 Q3: 31000 Q4: 31000 Justification: Storage requirements: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for JDFT-Surfaces
by accounts＠lcrc.anl.gov 07 Sep '17

07 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Kendra Letchworth-Weaver Project Name: JDFT-Surfaces Division: NST Project title: Joint Density-Functional Theory Investigations of Realistic Surface Structure Associated funding: LDRD/Named Fellow Other Systems: Carbon at Center for Nanoscale Materials (375,000 hours) Science: The surface structure of a surface or nanoparticle in contact with liquid is often different from the structure in vacuum. This structure may be probed experimentally by techniques such as X-ray reflectivity or pair distribution functions, however the results often require interpretation and validation. Using Joint-Density-Functional Theory, which enables quantum-mechanical calculations of a solute system (such as an surface) in contact with a liquid environment, we will make predictions of surface structure in liquid which will provide unique insights into related experimental studies. By avoiding the statistical sampling over configurations of the liquid required by classical and ab initio MD, JDFT enables rapid screening of surface terminations in liquid environment for agreement with experimental data. In addition, JDFT naturally provides surface energies in solution and as a function of voltage, allowing us to synthesize experimental and theoretical informat ion to determine surface structure under realistic electrochemical and growth conditions (as in doi:10.1021/jacs.6b03338). Project description: We plan to consider several systems within this project: 1. FY2017 The surface structure of geological materials in water, such as alumina or hematite, for direct comparison to X-ray reflectivity data from the group of Paul Fenter in CSE. We will perform AIMD to sample the surface alone while treating the liquid through JDFT. For alumina and hematite each, we will consider roughly 5 terminations in AIMD, requiring 10 ps of statistics for each. Parallelized over 9 NVidia C2075 GPUs, 1 ps of statistics for these surfaces requires roughly 24 hours of walltime. We expect that parallelizing over 6 faster Tesla K40 GPUs will yield the same walltime (2 materials x 5 terminations x 10 ps x 24 hours x 6 GPUs x 32 core hours=460800 hours) FY2018 We encountered some challenges with the simulations we planned to do in FY2017. First, we determined that in order to provide sufficiently accurate X-ray Reflectivity signals, the system sizes required were not feasible to study on gpu. We also determined that additional full AIMD calculations are required to benchmark the accuracy of JDFT for the liquid at the interface. Such calculations are not feasible within JDFTx, however they are possible in highly parallelizable Qbox. We request 5 terminations*2 functionals* 10 ps *1000 cores*2 hours/ps= 200,000 core hours on KNL to run AIMD simulations of the Al2O3 surface in water, using both van der Waals (optB88) and PBE functionals 2. FY2017 The surface structure of SrTiO3 during growth of Ruddlesden Popper stacking faults, for comparison to resonant X-ray reflectivity data from the group of Dillon Fong in MSD. Though there is no liquid present, these measurements were taken at high temperatures so AIMD is also required to account for thermal motion of the surface. The number of calculations required is similar to the values for the geological materials above. (1 material x 5 terminations x 10 ps x 24 hours x 6 GPUs x 32 core hours= 230400 hours) FY2018 We have concluded the study of homoepitaxial growth of SrTiO3, which will be submitted to Advanced Materials this Fall. Now, we turn to the heteroepitaxial growth of LaTiO3 on a SrTiO3 substrate, in an attempt to understand the different timescales of layer rearrangement experimentally observed on LTO deposition vs STO deposition. Here, the smaller unit cells were sufficient to run on the GPU. (1 material x 5 terminations x 10 ps x 24 hours x 6 GPUs x 32 core hours= 230400 hours) 3. FY2017 How the shape of IrO2 nanoclusters for solar water-splitting applications changes in the presence of liquid (in collaboration with Theory and Modeling group at CNM). We will also consider how the binding energy of oxygen at different adsorption sites on the nanoparticle changes when liquid is included. The initial low-energy IrO2 nanoparticle shapes will be determined by a genetic algorithm search using force fields to represent the atomic interactions. We will then consider approximately 10 low energy nanoclusters in liquid using JDFT. Finally, we will study approximately 10 oxygen binding sites on the lowest-energy nanocluster in solution. These calculations are not currently MPI parallelizable (though they are OpenMP and GPU parallelized to run on a single node and MPI parallelization may be implemented before the end of the allocation). These geometry optimization calculations will each take 1 week of walltime to run in vacuum and another week in liquid. (2 solvation states x 20 calculations x 7 days x 24 hours x 32 core hours=215040 hours) FY2018 These calculations proved to be unreasonably slow to converge on the older hardware at LCRC (many degrees of freedom required >100 ionic steps to minimize and they required too much memory to fit on the GPU). For this reason, much of the time requested in FY2017 went unused These calculations should run 10x faster on new KNL nodes, making them now feasible. (2 solvation states x 20 calculations x 7 days x 24 hours x 32 core hours=215040 hours) FY2017 The total production run hours above add to ~900,000 hours. FY2018 The total production run hours above add to ~650,000 hours We estimate using roughly an additional 150,000 hours (~20% of the requested allocation) for debugging and convergence tests. We will use the electronic structure code JDFTx, an open-source plane-wave DFT code co-developed by the PI and written in C++. It is available for download at http://jdftx.org. We require the following packages to be installed: • cmake (>=2.8) • g++ (>=4.6) • libgsl0-dev • libopenmpi-dev (or equivalent for other MPI distributions) • libfftw3-dev • libatlas-base-dev (provides blas and cblas) • liblapack-dev (alternatively, math operations using Intel’s MKL are possible) See http://jdftx.org/Compiling.html) for details. We will also need an older version of Qbox which supports OPTB88 van der Waals functionals to be installed and compiled. I will be working in collaboration with Maria Chan as well as my group members Fatih Sen and Arun Mannodi on this project for 4 team members in total. Industry partnership: Project URL: http://jdftx.org Current FY Hours Used: undetermined amount New FY Requested allocation: 800000 Q1: 200000 Q2: 200000 Q3: 200000 Q4: 200000 Justification: JDFTx can be run in either a hybrid GPU-MPI mode for smaller memory jobs or an OpenMP-MPI mode for larger memory jobs, offering excellent performance and scaling. GPU parallelization offers a speedup of 3-5x over a single CPU. The JDFT liquid experiences near-linear scaling when MPI-parallelized over orientations (typically 24-144 orientations are required) The electronic DFT experiences near-linear scaling when MPI-parallelized over k-points. Coulomb truncation algorithms implemented in the code also reduce the size of the simulation grid for JDFTx wavefunctions. As an example, scaling data for a small Al2O3 surface supercell (34 atoms, 9 k points) on NERSC Edison is provided below in vacuum. For the vacuum calculations, we perform initialization and a single electronic wavefunction minimization. Each MPI process was run with 12 OpenMP threads (2 processes per 24-core node). MPI Processes time 1 449 s 2 249 s 3 165 s 5 114 s 9 71 s The large allocation is requested because GPU calculations are charged double compared to CPU calculations, yet we will be able to utilize MPI parallelization over GPU’s. Alternatively, if the GPU is not available, we could MPI parallelize over CPU’s, though we would require the additional factor of 3-5 in walltime. JDFTx does run in parallel on KNL nodes (verified on NERSC Cori). Further optimization and benchmarking of the JDFTx software to run in hybrid MPI/threaded mode for KNL nodes is in progress. The ability to run on the KNL nodes will increase the system sizes available for study with JDFTx, yet will continue to run at speeds comparable or better than the GPU. Storage requirements: 1 TB Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for protein_e-wire
by accounts＠lcrc.anl.gov 07 Sep '17

07 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Oleksandr Kokhan Project Name: protein_e-wire Division: CSE Project title: Solar energy conversion in biomimetic molecular architectures Associated funding: This project is supported by three funding programs within the DOE-BES Division of Chemical Sciences, Geosciences, and Biosciences: i) Solar Photochemistry; ii) Photosynthetic Systems; iii) Energy Frontier Research Center: Argonne Northwestern Solar En Other Systems: N/A Science: This computational project will supplement our experimental work on construction and characterization of water-splitting catalysts and biomimetic molecular architectures for solar energy conversion. The first aim is to continue to use all-atom Molecular Dynamics (MD) simulations to predict conformations and structural dynamics of several photosensitizers covalently linked to PpcA, a multi-heme c-type cytochrome from Geobacter sulfurreducens. These results will facilitate interpretation of photo-induced kinetic electron transfer rates, Circular Dichroism (CD) spectroscopy, Heteronuclear Single Quantum Correlation (HSQC) NMR data as well as Small- and Wide-Angle X-ray Scattering (SAXS, WAXS). This will provide an insight into strategies for optimization of electron transfer rates within biomimetic hybrids. The second aim is to use MD to simulate equilibrium positions and dynamics of mutations needed to tune redox potentials of PpcA hemes. Our third goal is to use MD simulations to predict and elucidate structural mechanisms of ligand-driven protein oligomerization. Finally, we will start a new project focused on understanding of inter-domain static and dynamic interactions in several multiheme cytochromes and how they effect electron transfer rates through these “molecular wires”. Project description: We are developing biomimetic molecular architectures for efficient solar energy conversion using artificial and natural photosensitizers combined with natural and genetically engineered host systems capable to support long-lived charge-separated states and conduct charges away from the photosensitizers. We recently developed and characterized a series of 22 cysteine mutants of PpcA, a multiheme redox protein from Geobacter sulfurreducens, which demonstrate picosecond photo-induced electron transfer rates. Further, at least 2 attachment sites showed negligible activation energy barriers resulting in similar apparent reaction rates at 2.7 and 295K. These are remarkable results considering that neither picosecond photo-induced electron transfer rates nor activationless reaction rates were observed in comparable synthetic systems, though they are common in natural photosynthesis. However, any attempts in understanding how and why our experimental system repl icates Nature will require detailed structural information in order to interpret the observed kinetic data and to guide our future developments. Solution-state NMR data confirms tightly folded globular shape. However, obtaining full atomic structures is challenging due to a number of paramagnetic centers within the constructs. Obtaining high quality crystals in order to solve structures of labeled PpcA mutants so far has been elusive. The molecular dynamics simulations of in silico mutation followed by validating computational data with experimental X-ray scattering work and HSQC NMR provide us with a valuable tool to visualize structural changes, generate hypotheses about the mechanisms and factors controlling charge transfer and to develop experimental testing approaches. In the past we have developed CHARMM force field parameters for several photosensitizers and obtained promising preliminary results for some of PpcA mutants in their oxidized form linked to Ru(bpy)3 which closely matched X-ray scattering data. In FY2018 one area of our work will be focused on simulations of 14 PpcA mutants which do not show significant perturbations and unfolding in SAXS data. We will perform simulations on NAMD 2.12 with the most recent CHARMM36m force field. The latter is different from CHARMM27 used in our previous work and should produce even more reliable results and accomplish this goal within FY2018 unlike previously simulations spanning multiple funding cycles and forcing to use outdated force fields. Based on our previous experience, we expect that triplicate 300-500 ns all-atom MD simulations of selected PpcA biohybrids with explicit solvent will provide sufficient sampling of protein conformational space and help us to better rationalize the colle cted kinetic data on charge transfer rates and SAXS/WAXS scattering profiles as well as the observed perturbations in the NMR data. At the initial stages of the project we will focus on the systems with all three PpcA hemes either oxidized. For most interesting cases, we will also perform simulations with all hemes reduced. This is particularly interesting as it results in a significant change of the total protein change from +4 to +1. All MD simulations will be performed with the standard for protein MD conditions: nPT, PME electrostatics, explicit TIP3 water boxes sufficiently large to keep interaction energy negligible between protein mirror images. Though the system is relatively small with about 14,000 atoms, our benchmarks on Bebop show that we have about 20% speed up (30ps/core hour) per core when simulations are run on 1 Broadwell node in comparison with our benchmarks on Blues. Overall, we expect to perform simulations of about 20 different biohybrids in triplicate for 400ns. This will require 800,000 core hours. The second major goal is to expand our work on tuning heme potentials and electron transfer pathways within cytochromes using PpcA as a model system and introducing mutations in the heme environment. The mutations will be selected based on our previous experimental work and will be used in conjunction with ongoing crystallographic studies performed in collaboration with Raj Pokkuluri (BIO). The system size and computational approach will be the same as above. We anticipate about 15 unique systems which will require approximately 600,000 core hours. Our third goal is to switch to most recent CHARMM force field and to perform triplicate simulations for 4 different proteins (cyt c, cyt c4, PpcA, lysozyme). We will continue to use simulations with 4 copies of proteins and 8 photosensitizers. Previously we found that this is a good trade-off between the size and ability to see complex formation with binding/unbinding events happening at the time scale of 30-80 ns and remarkably well matching NMR data for PpcA, the only system which was accessible with NMR spectroscopy. These systems have about 100-120,000 atoms. They scale better than 90% with up to 4 Broadwell nodes and produce 4.2 ps/core-hour. We plan to run these simulations for 300-500ns to observe multiple binding and unbinding events. We expect to use 1,150,000 core hours. Our last aim is to simulate interdomain interaction in 4 domain (ABCD), 12 –heme GSU-1996 cytochrome. The structures of this entire protein (ABCD domains) as well as 2-domain constructs (AB, BC, CD) have been previously determined with X-ray crystallography. However, it is nearly certain that crystal packing has affected interdomain interactions and does not show an accurate heme-heme distances and interfacial interactions. We plan to use all-atom MD in combination with solution-state SAXS to study solution-state behavior of this “biological wire” spanning more than 100Å. We will perform triplicated simulations of the entire protein (59,000 atoms, 7.7 ps/core-hour) and all three 2-domain constructs (23,000 atoms, 16 ps/core-hour) for 400 ns. This work will require 390,000 core-hours. This project will help us to understand interfacial interactions between domains and their implications for inter-domain electron transfer. This will also help us to understand the consequ ences of protein truncation on domain dynamics which will ultimately allow us to build direction electron transfer chains sufficiently long to stabilize charge separates states at time scales comparable to native photosynthesis. Industry partnership: Project URL: http://blogs.anl.gov/solar-energy/ Current FY Hours Used: undetermined amount New FY Requested allocation: 2940000 Q1: 735000 Q2: 735000 Q3: 735000 Q4: 735000 Justification: NAMD 2.12 is a highly scalable molecular dynamics simulations code. While we still working on optimization on Bebop, with our currently compiled MPI code we already see about 20% improvement per Broadwell core over our benchmarks on Blues. For simulations involving multiple nodes we see better than 95% scaling with ~1,000atoms/core and ~80% with ~400atoms/core with network latency becoming a bottleneck beyond that ratio. Storage requirements: Default 1TB will be sufficient Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: ArOHE6
by accounts＠lcrc.anl.gov 07 Sep '17

07 Sep '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Albert Wagner Applicant's institution: ANL Applicant's division: CSE Project Name: ArOHE6 Project title: Excited OH Energy Relaxation in Ar Bath Gas Associated funding: BES – CSGB + US Army Research Office (for external colleagues [U. Missouri] analyzing computational results) Other Systems: Our group of ~10 computational chemists have accounts on ~660 haswell cores own by the group at the LCRC. These resources are largely reserved for electronic structure calculations, are well used, and as shown below are much slower than Bebop cores for the proposed task. Science: We want to investigate by molecular dynamics simulation if the isolated binary collision (IBC) approximation breaks down for a vibrationally excited OH in a thermal Ar bath gas. For low pressure bath gases, the energy relaxation of an excited molecule in the bath occurs through isolated binary collisions between the molecule and the bath gas. This means that all relaxation rates are directly proportional to pressure because pressure controls the isolated collision frequency. At high enough pressures the IBC approximation breaks down. This is relevant to advanced internal combustion engine design where pressures at their peak may lead to IBC breakdown. Past published studies of ours for excited CH3NO2 or HO2 in an Ar bath gas have shown this breakdown occurs at ~100 to 200 atm of Ar. Preliminary results on OH with an unrealistically but computationally efficient interaction potential with Ar show no evidence of change in pressure dependence up to 400 atm. We want to repeat this study with a realistic but computationally intense OH-Ar interaction potential. Project description: We will use LAMMPS with accelerators (OMP, INTEL) running NVE trajectories. We will look at 5 pressures (50, 100, 150, 200, and 400 atm) and do up to 1000 trajectories per pressure where each trajectory differs in its random initial positions and velocities. Each trajectory will follow 1000 Ar atoms and one excited OH molecule in a periodic box. The initial conditions have already been prepared for these 5000 trajectories by colleagues at U. Missouri. The simulation time length varies with pressure but 40 ns is a reasonable average based on ~50 preliminary trajectories calculated by colleagues at the U. Missouri. With a .05 fs time step, this amounts to 800M time steps for up to 5000 trajectories. A test case of 1M time steps has been run on blues and both types of cores of bebop. This 1002 atom simulation is small and the best time when run on all cores is ~55 seconds on blues haswell cores which translated to ~12 hrs per trajectory or 390 co re hours. However, when run on broadwell with LAMMPS INTEL accelerators but restricted to 1 core, the test case runs in ~125 seconds. The plan then is to partition bebop broadwell into 36 one core partitions and run simultaneously 36 production trajectories on a node which will complete in 27.8 hours or ~1000 core hours per 36 trajectories. A two trajectory simultaneous run confirms this approach. (Restarts are possible in LAMMPS and so this run can be broken up into smaller wall clock segments as needed.) At 1000 core hours per 36 trajectories, the cost of 1000 trajectories is ~28K core hours or 140K core hour for 1000 trajectories at 5 pressures. Tests on knights landing cores indicate noticeably longer times per trajectory. Tests on blues haswell cores indicate very much long times per trajectory. To accommodate significant differences in simulation times with respect to pressure, this proposal asks for 300K core hours. Industry partnership: none Project URL: Requested allocation: 300000 Q1: 200000 Q2: 100000 Q3: 0 Q4: 0 Justification: not applicable Storage requirements: 1 TB for this project is expected to be sufficient The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for perfopt
by accounts＠lcrc.anl.gov 07 Sep '17

07 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Prasanna Balaprakash Project Name: perfopt Division: MCS Project title: Optimization Algorithms for Empirical Performance Associated funding: DOE ASCR Other Systems: Mira, ANL: 100000 core hours Science: Our scientific objectives are: i) To advance the state-of-the-art in empirical performance tuning and adaptation of scientific applications on extreme scale architectures and thereby significantly increasing scientific productivity ii) To improve our understanding of large-scale empirical performance tuning iii) To develop novel mathematical optimization algorithms that exploits the characteristics of the search problem in performance tuning Project description: There are different strategies for performance model construction. Analytical approaches for performance modeling are cheaper in the sense that they don't suffer from the profiling and model generation cost as in the case of machine learning techniques. But analytical models are often difficult to construct for complex programs and they require detailed understanding of the underlying hardware and implementation of communication functions. Machine learning techniques come into play when it is difficult to construct an analytical model for an application. Models generated using machine learning techniques may not be as highly precise as an analytical model but still they help to understand the performance of an application that in turn help in optimizations such as auto tuning. In this research, we first try to build analytical models for communication of various application kernels. We use architecture-independent parameters as described in well known communication models such as LogGP family to build the analytical models. If the constructed analytical model is accurate enough in predicting the communication time, it saves the huge overhead in gathering training data for the application of machine learning algorithms. We will test the accuracy of the developed analytical models in predicting the communication time of several application kernels from the CORAL benchmarks. If the analytical model is not accurate enough (the error in prediction goes above acceptable range), we switch to gathering profile data and then generate models using machine learning techniques on that data. This hybrid approach gives us a balance between the complexity of building analytical models and the overhead of running expensive machine learning algorithms on huge amount of collected profiling data. Also the machine learning models are limited in the sense that we can't do extrapolation. Fitted analytical models come into play in that case. On Blues and Bebop, we will conduct an experimental study of communication models. A major benefit Blues brings to this research is the ability to test the impact of MPI implementations and interconnect techniques on Intel-based clusters. Eventually, we hope to deploy our techniques for improving the performance of scientific applications running on Blues. We estimate 100,000 core hours for our experimental study: evaluation of the strengths and limitations of LogGP models = 25,000 core hours; experimental analysis on multiple problem classes on different architectures = 30,000 core hours; empirical analysis for new approaches = 30,000 core hours; tests on parallel performance tuning approaches = 20,000 core hours; We will strongly involve in research dissemination activities such as writing scientific articles and giving seminars at Argonne and in conferences. In the the first and second quarter, our jobs will be 80% parallel. For the third and fourth quarter, we will have 100%. Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 105000 Q1: 25000 Q2: 30000 Q3: 30000 Q4: 20000 Justification: Storage requirements: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: BioFuel
by accounts＠lcrc.anl.gov 07 Sep '17

07 Sep '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Badri Narayanan Applicant's institution: ANL Applicant's division: MSD Project Name: BioFuel Project title: Metal carbide catalysts for upgrading bio-fuels Associated funding: EERE Other Systems: Science: Biomass holds tremendous promise as a sustainable source of carbon for synthesizing fuels, chemicals, and other carbon-based materials. In particular, the ubiquitous presence of oxygenate hydrocarbons in bio-mass derivatives has made the removal of O a critical research direction towards bio-fuel upgrading in the recent times. Two promising routes to reduce O content of bio-mass derivatives are (a) hydrodeoxygenation, involving C-O/C=O bond cleavage, and (b) decarbonylation or decarboxylation via C-C bond scission. Among these, the former (hydroxylation) has garnered lot of interest since it involves selective dissociation of C-O bonds while retaining the C-C chains; additionally, it produces H2O as by-product, which is more environment friendly than decarbonylation/decarboxylation that leads to CO2. Recently, metal carbides, e.g., Mo2C have been reported to be very effective in selectively catalyzing scission of C-O/C=O bonds without influencing the C-C bonds. Introduction of suitable dopants, e.g., Ni have been found to enhance selectivity, activity, and stability of these catalysts. Although experimental works on Ni-doped Mo2C indicate great promise of these catalysts for bio-fuel upgrading, a fundamental understanding of the role of dopants, including the type of dopants and their concentration in enhancing catalytic properties of Mo2C are not well known. Such knowledge is crucial to design new catalysts with exceptional selectivity, catalytic activity, and stability. In this project, we will employ total energy calculations, nudged elastic band barrier calculations, and charge distribution analysis in the framework of density functional theory (DFT) to understand the role of Ni, and its concentration on the catalytic activity of Mo2C catalysts. An important descriptor of the catalytic activity of Mo2C is the O* removal at Mo2C, since strong binding of O* on Mo2C can cause slow desorption of adsorbed atomic O and lead to catalyst poisoning. Hence, we will focus on binding energies of reaction intermediates during O* removal, namely O*, OH*, H2O*, and H * at different levels of Ni doping. This study will not only advance the fundamental knowledge of metal-carbide chemistry, but also enable computation-aided accelerated discovery of catalysts for bio-fuel upgrading. Project description: Our efforts will be focused on one commonly employed Mo2C surface, namely (001). We will employ density functional theory (DFT) calculations to compute the adsorption energies, as well as atomic charges for 4 adsorbents: (a) O*, (b) OH*, (c) H2O*, and (d) H* on Mo2C (001) doped with Ni at 5 different concentrations in the range 0--8% by weight. All the calculations will be performed on a Ni-Mo2C (001) slab using plane-wave DFT package VASP; the Brillouin zone will be sampled with 4 x 4 x1 Monkhorst-Pack grid, and plane wave energy cutoff of 400 eV will be used. The Perdew-Burke-Ernzerhof pseudopotentials will be employed for exchange correlations within the generalized gradient approximation. The atomic positions for the dopant Ni atoms, as well as the adsorbents are known from our recent computational work. Each of these simulations is expected to take 150 hours on 96 cores (i.e., 3 nodes on Bebop). So, we need 5 (concentrations) x (4 (adsorbents) + 1(bare slab)) x 96 (cores) x 150 (hours) = 360,000 core hours. Next, we will compute the energy barriers for 3 different reactions (a) O* + H* ↔ OH* +*, (b) OH* + H* ↔ H2O* +*, and (c) OH* + OH* ↔ H2O* +O* at each dopant concentration using climbing image nudged elastic band approach. Each of these calculations is expected to take about 96 hours on 96 cores. So, we require 5 (concentrations) x 3 (reactions) x 96 (cores) x 96 (hours) = 138,240 core hours. In all, we require a total of 498,240 core hours Industry partnership: Project URL: Requested allocation: 498240 Q1: 124560 Q2: 124560 Q3: 124560 Q4: 124560 Justification: Storage requirements: 1 TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for SnI4
by accounts＠lcrc.anl.gov 07 Sep '17

07 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Michael Hu Project Name: SnI4 Division: XSD Project title: Modeling of the structure and vibrational dynamics of materials at High Pressures Associated funding: DOE and NSERC (Canada) Other Systems: Science: Nuclear resonant inelastic x-ray scattering (NRIXS) is a synchrotron-radiation-based method that finds a wide range of applications in condensed matter physics, materials science, high pressure research, geosciences, and biophysics. Electronic structure calculations are to be used to assist the assignments of vibrational density of states (VDOS) obtained from high pressure NRIXS and diffraction experiments. Several phase transition systems will be studied with the combination of simulations and x-ray experiments done primarily at APS. Molecular dynamics can also be used to simulate the spectra measured with NRIXS. This approach will help better understand the lattice dynamics of the systems under study. It will broaden the applications of NRIXS method to more complex and more realistic samples. There are three objectives for this renewal request. [1] SnI4 This is a continuation of the previous project. Using previous CPU allocation, we have performed First Principles Molecular Dynamics calculations and correctly reproduced the high pressure transformation sequence, viz. an ambient crystalline phase to an intermediate crystal structure prior to amorphization and subsequent recrystallization to a face-center cubic (FCC) phase at 64 GPa. The calculated Sn vibrational density of states (VDOS) were in good agreement with experiment available at the time below 60 GPa. The results were published in J. Chem. Phys., 143, 164508 (2015). However, recent experiments extended beyond 60 GPa have found that the Sn VDOS in the FCC phase differ significantly from theoretical predictions. The reason is not obvious. We propose to investigate the dynamics of the high pressure phase in more detail in order to reveal the cause of this discrepancy. [2] SnSe SnSe is a potentially high-efficiency thermoelectric material with a fairly high figure of merit (ZT). It has an abnormally low thermal conductivity, a sign of large enharmonic motions. We have studied the atomic vibrations and computed the phonon band structure using lattice dynamics method and the temperature dependence of the Sn projected density of states with large scale molecular dynamics simulations. Recently temperature dependent and high pressure inelastic scattering data became available. We will perform calculations to better interpret these data, and to understand the effect of anharmonicity. [3] Sn-Fe alloys Recently Sn in Fe alloys were studed with NRIXS at 30ID of APS in a geochemical study of isotope fractionation. The aim of this research is to make Sn isotopes the new probe of the early evolution of the Moon, the Earth and Mars. Siderophile (iron-loving) elements provide clues to the conditions under which terrestrial planets differentiated into metallic cores and silicate mantles. The concentrations of such elements (e.g. Ni, Co, W, Sn) in the Earth’s mantle have been set by metal–silicate equilibrium in an early magma ocean. But the thermodynamic conditions presiding to this differentiation (P, T, fO2) are a matter of intense debate. We will use MD simulations to interpret measured NRIXS spectra and build models of its lattice dynamics and isotope fractionation. Project description: Static and dynamic First-principles electronic structure calculations will be performed to characterize the structure and vibrations of both the SnI4 and SnSe. The VASP code will be used primarily for geometry optimization. In previous work molecular dynamics were also performed with the VASP code. From our experience, on a small system consisted of 32 SnI4 molecules in a cubic box(160 atoms) using only one k-point in the simulation, the scaling of VASP was quite reasonable up to 64 CPUS. In the new study, we wish to study to investigate system size effect to the vibrational frequencies. For this purpose, we wish to use the update Qbox code implemented with the ultrasoft pseduopotential (QBall). QBall/Qbox have been demonstrated to scale well on large computer cluster,s particularly on the IBM Blue/gene. Our first goal is to swtup QBall on the cluster and test the scalabiity on increasing the system size incrementally. In the primitive cell of SnI4 there are 4 molecules (20 atoms), we believe a model consisted of 3x3x4 or 4x4x4 replicate of the unit cell should be feasible. Since the Sn and I atoms in the high pressure FCC phase is positionally disordered, we shall employed the special quasi-random structure method (SQS) to generate the model. This is different from our previous study in which the structure obtained from the compression of the low pressure structure with molecular dynamics calculations was used. We expect the SQS will give a better representation of the disorder solid. For the Sn-Fe alloy systems, so far we have completed molecular dynamics calculations of the system at 300K using spin-unrestricted calculations. The computed Fe and Sn density of states are in good agreement with experiment. However, Fe is in the high spin state under normal conditions, it essential to perform spin-unrestricted calculations or, perhaps including self-interaction correction through the Hubbard model. Initial calculations show self-convergence is very poor once spin-unrestricted Hamiltonian was employed. We are currently investigating if the convergence can be improved by including on-site repulsion on the Fe atoms. Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 490000 Q1: 120000 Q2: 120000 Q3: 120000 Q4: 130000 Justification: Storage requirements: The MD trajectory files are quite large. From previous experience for each run, the total size of all output files are close to 1 Gb. We request 1 TB of disk storage Thank You, The LCRC Accounts System

1 0