Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Olle Heinonen Project Name: Meso Division: PSE Project title: Mesoscale computational materials science Associated funding: LDRD, BES (FWP 59001-00-105), CHiMaD (NIST), BES (Computational Materials Hub MiCCom) Other Systems: serafina cluster (MSD), BG/Q, CNM cluster Science: - Simulations of mesoscale coupled elastic/ferroic systems - Oxygen defect motion in metal/oxide/metal heterostructures - Magnetization behavior and dynamics in nanostructured magnetic systems. Project description: This project has three thrusts: (i) development of computational codes and algorithms for mesoscale modeling of inhomogeneous materials systems and applications of these codes, (ii) modeling of magnetic micron- and nano-scale systems, and (iii) first-principle and molecular dynamics modeling of metal/oxide/metal heterostructures. In the first thrust, we are developing scalable finite element codes for coupled elastic-ferroic systems, as well as charged particles immersed in a dielectric fluid subjected to electric fields. The code developments build on libraries such as libmesh and PETSc to achieve portable, scalable and highly efficiently parallelized codes. In particular, we are developing codes that combine fast-multipole methods with boundary element methods (FMM-BEM)to calculate long-range electrostatic and magnetostatic fields from bound and free surface charges. Within the MOOSE framework, we are also developing solvers for coupled ferroelectric/ferromagnetic-elastic systems. We will in the coming year integrate the FMM-BEM method with the MOOSE framework and apply the codes to study ferroelectric or ferromagnetic thin films and nanoparticles, as well as inhomogeneous composite multiferroic systems. We are in contact with a number of experimental groups with which we expect to collaborate. We a re also developing fast generalized geometry Ewald methods to coupled particles and long molecules, such as polymers, to fluids. This latter part is in close collaboration with Juan de Pablo and his group. The work in this first thrust will also be integrated into codes under the MiCCom computational materials hub led by Giulia Galli. The codes developed under this thrust are also massively scalable to leadership-class computer platforms. The programming language for these developments is C++. As we move from development to production phases we expect to run on up to 248 cores on fusion and blues. The MOOSE framework has been shown to scale almost linearly up to some 100k cores on BG/Q without any BG/Q-specific optimization. For the FMM-BEM method we have already shown linear scaling with the number of cores (using hybrid MPI-threading) to hundreds of cores. In the second thrust we are modeling static and dynamic magnet configurations in a variety of magnetic systems. This work is in collaboration with a number of experimental groups at Argonne and around the world. The workhorse of this thrust is a homemade micromagnetic code, or set of codes, in FORTRAN90 parallellized using OpenMP in addition to mkl libraries (in particular mkl dft); they only run on one node at a time but have been optimized and are highly efficient, typically running at 95% efficiency on eight to 16 cores. These runs typically consume one node on fusion or blues (all cores on the node) for one to 30 wall-hours. In thrust (iii), we are predominantly using the SIESTA/smeagol (a development version) and FHI-aims density functional theory based codes. Both have been optimized for blues/fusion and we can routinely run on up to a few hundred cores (up to some tens of thousand cores on BG/Q) using hybrid parallelization (smeagol) or just MPI tasks (FHI-aims). We plan to model finite-bias conductance in TiN-HfO2-Ta-TiN structures using non-equilibrium Green's functions (NEGF) in smeagol on systems containing 600-1000 atoms (we have already tested the performance on blues); the largest runs will use up to 512 cores for a few days wall time on blues. We will also do nudged elastic band calculations of oxygen diffusion barriers in the HfO2 using SIESTA/smeagol with about the same requirements for the largest runs. We will also combine molecular dynamics (MD) simulations using LAMMPS with NEGF; from an MD perspective, runs with some 1000 atoms are very small and the time required is negligible compared to the requirements for smeagol. We expect there to be about eight members in this project (PI, two co-PIs, four post-docs and one graduate student). Industry partnership: N/A Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 600000 Q1: 150000 Q2: 150000 Q3: 150000 Q4: 150000 Justification: We have worked extensively with one of the smeagol developers, Ivan Rungger (who is also one of the co-PIs) to optimize SIESTA/smeagol and we now have very good scaling for up to some tens of thousand cores. FHI-ames has been optimized on blues with help from ALCF/LCRC staff and scales very well for up to some hundreds of cores (the larges we have tried). For the development under the MOOSE framework, we constantly check scaling of the developed codes to make sure they scale basically linearly to hundreds of cores before we start production runs. We wasted some core-hours FY2015 by running some codes (especially FEniCs) inefficiently; I have since gone after post-docs to make sure the optimize preconditioners and MPI/thread balance, and also spend more time on preparation using small number of cores or desktops/laptops, before starting to consume larger resources fusion/blues. The allocation request is large, not because we will do a number of runs that will c onsume a very large number of cores, but because we (a) spend a lot of time on code development, and (b) we have a relatively large number of project members who work extensively on blues/fusion. Storage requirements: 2 TB. We have already used up 1TB. I am constantly asking project members to do house cleaning, delete obsolete data, compress and archive data to be kept. Thank You, The LCRC Accounts System