Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Jeff R. Hammond Project Name: batteryQC Division: LCF Project title: Quantum chemical simulations of batteries and related algorithms and software tools Associated funding: Argonne LDRD, Chemistry Exascale Codesign Center Other Systems: Intrepid/Surveyor: Blue Gene/P at Argonne (INF), Chinook: Opteron/IB at PNNL (8M), Jaguar: Cray XT5 at ORNL (?), Kraken: Cray XT5 at NICS/Teragrid (1M), Ranger: Opteron/IB at TACC/Teragrid (1M), Lincoln: Tesla/Xeon/IB at NCSA/Teragrid (100K), Dirac: Fermi/Nehalem/IB at NERSC (?), Hopper: Cray XE6 at NERSC (?), Palu: Cray XE6 at CSCS (?) Science: (1) quantum chemistry for metal-oxide systems related to battery science (2) statistical mechanics methods for non-equilibrium systems (3) communication runtime systems for HPC networks Project description: Accurate quantum chemistry methods, particularly coupled-cluster theory (CC), will be applied to metal oxide systems in order to understand the chemical properties of batteries. The CC method known CCSD(T) is often called the "gold standard" of quantum chemistry and provides the most accurate results available for a polynomial-scaling algorithm when applied to most chemical systems. In contrast, density-functional theory (DFT) cannot be used reliable for battery-like systems since localized electronic defects create problems for most of the approximate exchange-correlation functionals used today. This project seeks to calibrate quantum chemistry methods for battery-like systems and develop software to enable Argonne's computational investigations in battery science. The second component of this project is the development of massively-parallel non-equilibrium umbrella sampling (NEUS) algorithms (see http://code.google.com/p/parallelsampling). Alex Dickson and Aaron Dinner have developed a theory for this and Alex and Jeff have been implementing this algorithm in parallel using the Global Arrays toolkit. Both software development and applications will be performed using this allocation. We have a poster at Supercomputing 2010 poster accepted on the preliminary results of this work and have submitted an Argonne-UChicago Strategic Collaborative Initiative proposal to continue this work. We have been developing a new communication runtime system for Global Arrays and PGAS languages called OSPRI (formerly known as A1 - Argonne 1-sided - internally, see https://trac.mcs.anl.gov/projects/a1/). The implementation for Blue Gene/P over DCMF is essentially complete and we are now working on an implementation for commodity systems using MPI plus network-specific extensions (e.g. read-modify-write using IB Verbs). Because OSPRI is a drop-in replacement for ARMCI, we will be conducting performance tests of NWChem and Global Arrays using the new runtime. Experiments Planned: We continue to study lithium oxide clusters using coupled-cluster methods in collaboration with the Curtiss group at Argonne and to tune NWChem for Fusion for this and other purposes. The site installation of NWChem is maintained by the PI of this project and testing is conducted as part of this project, which benefits a number of researchers at Argonne. The NEUS experiments are designed by the Dinner group and involve: (1) evaluating the correctness of the NEUS algorithms for various systems, (2) optimizing hybrid MPI-OpenMP parallelism and (3) measuring parallel scalability over IB in comparison to BG/P. All OSPRI experiments are communication benchmarks of one-sided communication from the GA/ARMCI test suite as well as strong-scaling tests of NWChem. Programming Model and Scalability Issues: NWChem relies upon the Global Arrays (GA) library to provide a PGAS programming model which includes one-sided put/get/accumulate and dynamic load-balancing. Scalability issues appear only at the onset of data-starvation, but this is not critical, as efficient strong-scaling can be achieved over more than an order of magnitude variation in node-count, and greater strong-scaling could be achieved if low-end data was available (jobs are memory-bound). Software Requirements: NWChem requires MPI (C bindings only), the OpenIB stack, F77 and C compilers, and an optimized BLAS library. Performance Results and Efficiency of the Code: The NWChem CCSD(T) code hit 92% of theoretical peak on ~1000 Itanium CPUs on the old PNNL supercomputer and ~50% of theoretical peak on close to 100K CPUs of Jaguar. Flop-rate on BlueGene/P is 20-50% of peak due to the small number of registers, other memory limitations, and the absence of true RDMA capability on the BGP interconnect. This is a key motivation for using Fusion. Algorithmic developments are expected to improve the performance of non-BLAS3 operations, such as atomic integral generation. Job Characterization: We expect to run 12-24 hour jobs on 64-256 nodes for production runs and 1-2 hour jobs on 1-16 nodes for software development. While the fat nodes will be helpful in some cases, we are not dependent on them since NWChem scales to the full machine and no data is replicated. Other Information: The expected number of project members is approximately five. Jeff Hammond works on all aspects of the project. Alex Dickson works on NEUS. Jim Dinan will use this allocation for developing OSPRI if necessary. Jeff's summer interns use this project as necessary, but their consumption is nominal. Publications: Over the past six months, this project was used to generate results for the following papers: R. S. Assary, P. C. Redfern, J. R. Hammond, J. Greeley and L. A. Curtiss, Chem. Phys. Lett., accepted (2010). "Predicted Thermochemistry for Chemical Conversion of 5-Hydroxymethyl Furfural." R. S. Assary, P. C. Redfern, J. R. Hammond, J. Greeley and L. A. Curtiss, J. Phys. Chem. B, 114, 9002 (2010). "Computational Studies of the Thermochemistry for Conversion of Glucose to Levulinic Acid." The work on NEUS and OSPRI/A1 has been the subject of oral presentations and posters; conference and journal papers will be submitted in the next six months. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 200000 Justification: NWChem completely parallelized and scales to many thousands of CPUs. The communication mechanism, ARMCI, has been extensively optimized for Infiniband and provides one-sided performance very close to the hardware capacity. The particularly methodology to be used for this project, CCSD(T), is implemented two different ways in NWChem, one of which scales at >90% efficiency from 1024 to 6144 CPUs on the Chinook supercomputer, which is very similar to Fusion, except larger. The other implementation hit ~0.5 PF on 96K cores of JaguarPF and is the basis for a Gordon Bell Finalist by another group of NWChem developers. The serial kernel in CCSD(T) is DGEMM, which should hit 90% of machine peak using Intel MKL. Performance analysis on both Surveyor and Chinook indicate that DGEMM is >50% of the wall time for large jobs and the rest can be tuned to achieve ~10% of theoretical peak. Hence, these should run at ~50% of theoretical peak (50%*90%+50%*10%=50%). The motivation to use Fusion instead of other machines I have access to is that most jobs are memory-bound, so BlueGene/P is not ideal, and the operating systems on both Chinook and Jaguar are unreliable at scale. The NEUS code scales from 1 to 8K cores of BG/P with 99% efficiency. The hybrid parallelism using OpenMP is near perfect on BG/P within a node. On Fusion, we expect less efficient thread parallelism but nonetheless, NEUS will run on the entire Fusion machine if necessary. OSPRI aka A1 has been tested extensively on BG/P and is, by design, almost scale-free. Thank You, The LCRC Accounts System