March 2017 - Allocations-admins - mailman3.cels.anl.gov

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Roberts, John E. 10 Mar '17

10 Mar '17

Looks like this one slipped through the cracks. What should we approve? Thanks. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 3/3/17, 10:06 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: linyun (Linyun Liang) Project: Meso_Nuclear Title: Gas bubble kinetics in an irradiated U-Mo using a mesoscale simulation approach Description: A mesoscale simulation approach is developed to study the gas bubble evolution kinetics in irradiated U-Mo alloy fuels. Density functional theory is first utilized to predict the defect formation energies, the Xe-U-Mo interfacial energy, and the elastic constants of U-Mo alloys, among other quantities. These obtained material properties are then incorporated into a mesoscale model to study the kinetics of gas bubbles in U-Mo under irradiation. By developing a new free energy model of Xe gas bubble based on the van der Walls equation of state, the gas bubble pressure can be estimated. The effects of vacancy and self-interstitial atom (SIA) diffusion on the gas bubble pressure will be further investigated and validated. The events employed in the simulation are the absorption and emission of a vacancy, a SIA, and an interstitial U by a single Xe bubble, while an event associated with SIA loops or substitutional U is neglected here for simplicity. We will systemat ically investigate the effect of Xe, vacancy, and SIA concentration, fission defect generation and their recombination, sink strength, grain boundaries, and elastic interaction on the growth kinetics of gas bubble. The swelling of U-Mo due to fission gas bubble will be simulated and compared to experimental observations. Our study aims to provide insight into understanding of the structural damage in irradiated U-Mo materials. In our phase field model, we have four highly non-linear, coupled partial differential equations which must be solved numerically. In the meaning time, the elastic equilibrium equation is also included in each time step, which requires to obtain the elastic solution at the equilibrium state for each time step and can significantly increase the cost of the simulation time. And also in order to avoid the partial loss of interfacial energy, we will extend our model to three dimensional system. Therefore, to effectively solve this problem, we will need about 150,000 core hours including the 6,000 core-hour to debug and test the algorithm and code and 144,000 core-hours (4 cores/node * 4 nodes * 10 hours of run time * 800 different runs) for the nucleation and growth of gas bubbles calculations. We will use our developed FFTW code to solve the problem. By using the Fusion and Blues computing resources will greatly accelerated the code development and calculations. In conclusion, the total core-hours requested for this project throughout the FY2016 will be 150,000 core-hours. Fig. 1 Simulated results of gas bubbles nucleation and growth under the irradiated U-7Mo. Current: undetermined amount Justification: Requested: 300000 A specific reason has been given: I have run out of my core-hours for this project. I request to have additional core-hours from now. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Roberts, John E. 08 Mar '17

08 Mar '17

Time granted. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 3/8/17, 8:06 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of Bair, Raymond A." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of rbair(a)anl.gov> wrote: Lets add the 100K requested for March. Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 3/7/17, 3:29 PM, "Heinonen, Olle G." <heinonen(a)anl.gov> wrote: Hi Ray, Well, we've been burning about 100k core-hours per month. If I can get that, that would be great. I think we can get by on 50k core-hours. Thanks, Olle Olle Heinonen, Materials Scientist Group Leader, Condensed Matter Theory Materials Science Division Argonne National Laboratory, bldg 200, 9700 South Cass Ave. Lemont, IL 60439 heinonen(a)anl.gov, Tel +1 630 252 4877 ________________________________________ From: Bair, Raymond A. Sent: Tuesday, March 07, 2017 2:38 PM To: Heinonen, Olle G. Subject: Re: [allocations-admins] [LCRC Accounts] Project Allocation Request Dear Olle, April starts the 2nd half of FY17, and you will be getting the second half allocation then. How much more time do you need between now and March 31? Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 3/6/17, 11:11 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: heinonen (Olle Heinonen) Project: Meso Title: Mesoscale computational materials science Description: This project has three thrusts: (i) development of computational codes and algorithms for mesoscale modeling of inhomogeneous materials systems and applications of these codes, (ii) modeling of magnetic micron- and nano-scale systems, and (iii) first-principle modeling of metal/oxide/metal heterostructures and complex oxides. In the first thrust, we are developing scalable finite element codes for coupled elastic-ferroic systems, as well as charged particles immersed in a dielectric fluid subjected to electric fields. The code developments build on libraries such as libmesh and PETSc to achieve portable, scalable and highly efficiently parallelized codes. In particular, we are developing codes that combine fast-multipole methods with boundary element methods (FMM-BEM)to calculate long-range electrostatic and magnetostatic fields from bound and free surface charges. Within the MOOSE framework, we are also developing solvers for coupled ferroelectric/ferromagnetic-elastic systems. We will in the coming year integrate the FMM-BEM method with the MOOSE framework and apply the codes to study ferroelectric or ferromagnetic thin films and nanoparticles, as well as inhomogeneous composite multiferroic systems. We are in contact with a number of experimental groups with which we expect to collabora te. The work in this first thrust is integrated into codes under the MICCoM computational materials center led by Giulia Galli. The codes developed under this thrust are also massively scalable to leadership-class computer platforms. The programming language for these developments is C++. As we move from development to production phases we expect to run on up to 248 cores on fusion and blues. The MOOSE framework has been shown to scale almost linearly up to some 100k cores on BG/Q without any BG/Q-specific optimization. For the FMM-BEM method we have already shown linear scaling with the number of cores (using hybrid MPI-threading) to hundreds of cores. In the second thrust we are modeling static and dynamic magnet configurations in a variety of magnetic systems. This work is in collaboration with a number of experimental groups at Argonne and around the world. The workhorse of this thrust is a homemade micromagnetic code, or set of codes, in FORTRAN90 parallellized using OpenMP in addition to mkl libraries (in particular mkl dft); they only run on one node at a time but have been optimized and are highly efficient, typically running at 95% efficiency on eight to 32 cores. These runs typically consume one node on fusion or blues (all cores on the node), including the msd queue's Haswell nodes for one to 30 wall-hours. In thrust (iii), we are using the SIESTA/smeagol (a development version), quantum espresso (QE) and FHI-aims density functional theory based codes. All have been built and optimized for blues/fusion and we can routinely run on up to a few hundred cores (up to some tens of thousand cores on BG/Q) using hybrid parallelization (smeagol) or just MPI tasks (QE and FHI-aims). We are continuing to model electronic structure and transport properties of oxide heterostructures and complex oxides. The focus is shifting a bit FY2017 since this thrust will be aligned with the BES-funded computational materials center (led by ORNL). We expect there to be about seven members in this project (PI, two co-PIs, three post-docs and one graduate student). Current: undetermined amount Justification: The codes we will use have all been tested and optimized and run efficiently on blues. Requested: 524503 A specific reason has been given: We have run out of allocation. The project is servicing three different BES-funded projects (MICCoM, condensed matter theory FWP in MSD, and the computational materials science center led by Paul Kent at ORNL but with a large presence at ANL), and also the CHiMaD center of which I am a PI. We need another 500k core hours for FY2017 - we try to run as much as possible on the msd queue but it only has 30 nodes and we have to run on blues and the haswell queue, too. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - XRayFuelSpray
by Roberts, John E. 08 Mar '17

08 Mar '17

Time granted. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 3/8/17, 8:05 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of Bair, Raymond A." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of rbair(a)anl.gov> wrote: Lets add the 20K requested for March. Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 3/7/17, 4:19 PM, "Christopher F. Powell, Ph.D." <powell(a)anl.gov> wrote: Ray, We've got a paper due at the end of March, and 15-20K hours would let us finish that up on time. thanks, chris On 3/7/2017 3:00 PM, Bair, Raymond A. wrote: > Dear Chris, > > April starts the 2nd half of FY17, and you will be getting the second half allocation then. How much more time do you need between now and March 31? > > Regards, > > Ray > > > ----------------------------------------- > Ray Bair > Argonne National Laboratory > and the University of Chicago > > > On 3/3/17, 11:43 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: > > Hello, > > A change in allocation has been requested: > > Requester: cpowell (Chris Powell) > Project: XRayFuelSpray > Title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics > Description: The cavitation and diesel spray modeling sub-tasks are being performed using the OpenFOAM framework. These tasks will consume the bulk of the requested core hours. A number of state-of-the-art numerical solvers have been implemented in OpenFOAM; namely incompressible-liquid and fully compressible homogeneous relaxation cavitation models and more recently a fully compressible homogeneous relaxation cavitation model which includes non-condensable gas modeling (HRMFoam). This code can handle both cavitating as well as flash boiling flows. These codes were developed by Schmidt et al at the University of Massachusetts-Amherst, and we continue to collaborate with them on code development and efficiency improvements. The HRMFoam solver has been tested for scalability and shown good performance on up to 300 processors. We typically run jobs with 64 to 128 processors. > > The diesel spray sub-task in OpenFOAM studies turbulent, aerodynamic, and hybrid spray atomization sub-model formulations for high-pressure fuel sprays via customized sub-models. The target sprays for simulation will be the Engine Combustion Network (ECN) non-vaporizing Spray A and Spray D conditions. LCRC resources will enable GA Tech to rapidly assess model formulations and sensitivities over a wide range of liquid Weber and Reynolds number conditions, probing the relevance of turbulence and aerodynamic atomization mechanisms at limiting conditions. > > The combusting spray sub-task in OpenFOAM investigates the coupling of spray sub-models and turbulence-chemistry interactions for spray combustion through the development and implementation of a Large-Eddy Simulation (LES) framework. The target sprays for simulation will be the Engine Combustion Network (ECN) reacting Spray A condition. LCRC resources facilitate the simulation of multiple LES realizations in order to obtain reliable statistics for predicted spray and combustion parameters, which is essential for comparison against experimental measurements from APS and GA Tech. These comparisons will enable an in-depth study of both the instantaneous and average coupling between spray-induced turbulence structures and the resultant chemistry and flame evolution. > > The tasks described above will be run on 64 to 128 processors and will consume the bulk of the allocation requirement. Due to the conclusion of several sub-tasks from the previous FY, our allocation requirement for FY17 is reduced. > Tomographic reconstruction of APS data will be conducted using the TomoPy software package developed at Argonne. We run single-node jobs to efficiently perform multiple reconstructions of large (hundred-GB) datasets; these typically require less than 100 core hours each, but have high RAM requirements. This task will benefit substantially from the availability of the more powerful Haswell nodes and we are investigating the possibility of purchasing dedicated nodes on the new Bebop machine for this purpose. > > We have also successfully compiled a Paraview server on Blues which allows us to rapidly visualize the results of our simulations and reconstructions without having to wait to download the data to our local computer. This allows us to achieve rapid turnaround of results during beamtime. This is a significant aid to the experimental program. The server runs typically on a single Blues ‘biggpu’ node (16 cores) which allows us to open simulation datasets that are much larger than the RAM available on our desktop computers. This task requires relatively few core hours, since the server is only used for a few hours at a time. > Current: undetermined amount > Justification: Due to the conclusion of several sub-tasks, and plans to purchase dedicated nodes, our allocation requirements for FY17 are much less than in the previous year, and will not exceed 0.5M core hours. > > Requested: 20000 > > A specific reason has been given: > In our collaboration with Georgia Tech, we are working on fuel injection simulations for an upcoming publication. Additional CFD runs were required for grid convergence, and more CPU hours are required to complete this work. > > This needs to be approved and the final allocation amount decided upon. > > Thank You, > The LCRC Accounts System > > -- Christopher F. Powell, Ph.D. Center for Transportation Research Argonne National Laboratory 9700 South Cass Avenue Bldg 362, Room C217 Argonne, IL 60439 email: powell(a)anl.gov phone: (630) 252-9027 fax: (630) 252-3443 _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 08 Mar '17

08 Mar '17

Lets add the 100K requested for March. Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 3/7/17, 3:29 PM, "Heinonen, Olle G." <heinonen(a)anl.gov> wrote: Hi Ray, Well, we've been burning about 100k core-hours per month. If I can get that, that would be great. I think we can get by on 50k core-hours. Thanks, Olle Olle Heinonen, Materials Scientist Group Leader, Condensed Matter Theory Materials Science Division Argonne National Laboratory, bldg 200, 9700 South Cass Ave. Lemont, IL 60439 heinonen(a)anl.gov, Tel +1 630 252 4877 ________________________________________ From: Bair, Raymond A. Sent: Tuesday, March 07, 2017 2:38 PM To: Heinonen, Olle G. Subject: Re: [allocations-admins] [LCRC Accounts] Project Allocation Request Dear Olle, April starts the 2nd half of FY17, and you will be getting the second half allocation then. How much more time do you need between now and March 31? Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 3/6/17, 11:11 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: heinonen (Olle Heinonen) Project: Meso Title: Mesoscale computational materials science Description: This project has three thrusts: (i) development of computational codes and algorithms for mesoscale modeling of inhomogeneous materials systems and applications of these codes, (ii) modeling of magnetic micron- and nano-scale systems, and (iii) first-principle modeling of metal/oxide/metal heterostructures and complex oxides. In the first thrust, we are developing scalable finite element codes for coupled elastic-ferroic systems, as well as charged particles immersed in a dielectric fluid subjected to electric fields. The code developments build on libraries such as libmesh and PETSc to achieve portable, scalable and highly efficiently parallelized codes. In particular, we are developing codes that combine fast-multipole methods with boundary element methods (FMM-BEM)to calculate long-range electrostatic and magnetostatic fields from bound and free surface charges. Within the MOOSE framework, we are also developing solvers for coupled ferroelectric/ferromagnetic-elastic systems. We will in the coming year integrate the FMM-BEM method with the MOOSE framework and apply the codes to study ferroelectric or ferromagnetic thin films and nanoparticles, as well as inhomogeneous composite multiferroic systems. We are in contact with a number of experimental groups with which we expect to collaborate. The work in this first thrust is integrated into codes under the MICCoM computational materials center led by Giulia Galli. The codes developed under this thrust are also massively scalable to leadership-class computer platforms. The programming language for these developments is C++. As we move from development to production phases we expect to run on up to 248 cores on fusion and blues. The MOOSE framework has been shown to scale almost linearly up to some 100k cores on BG/Q without any BG/Q-specific optimization. For the FMM-BEM method we have already shown linear scaling with the number of cores (using hybrid MPI-threading) to hundreds of cores. In the second thrust we are modeling static and dynamic magnet configurations in a variety of magnetic systems. This work is in collaboration with a number of experimental groups at Argonne and around the world. The workhorse of this thrust is a homemade micromagnetic code, or set of codes, in FORTRAN90 parallellized using OpenMP in addition to mkl libraries (in particular mkl dft); they only run on one node at a time but have been optimized and are highly efficient, typically running at 95% efficiency on eight to 32 cores. These runs typically consume one node on fusion or blues (all cores on the node), including the msd queue's Haswell nodes for one to 30 wall-hours. In thrust (iii), we are using the SIESTA/smeagol (a development version), quantum espresso (QE) and FHI-aims density functional theory based codes. All have been built and optimized for blues/fusion and we can routinely run on up to a few hundred cores (up to some tens of thousand cores on BG/Q) using hybrid parallelization (smeagol) or just MPI tasks (QE and FHI-aims). We are continuing to model electronic structure and transport properties of oxide heterostructures and complex oxides. The focus is shifting a bit FY2017 since this thrust will be aligned with the BES-funded computational materials center (led by ORNL). We expect there to be about seven members in this project (PI, two co-PIs, three post-docs and one graduate student). Current: undetermined amount Justification: The codes we will use have all been tested and optimized and run efficiently on blues. Requested: 524503 A specific reason has been given: We have run out of allocation. The project is servicing three different BES-funded projects (MICCoM, condensed matter theory FWP in MSD, and the computational materials science center led by Paul Kent at ORNL but with a large presence at ANL), and also the CHiMaD center of which I am a PI. We need another 500k core hours for FY2017 - we try to run as much as possible on the msd queue but it only has 30 nodes and we have to run on blues and the haswell queue, too. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - XRayFuelSpray
by Bair, Raymond A. 08 Mar '17

08 Mar '17

Lets add the 20K requested for March. Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 3/7/17, 4:19 PM, "Christopher F. Powell, Ph.D." <powell(a)anl.gov> wrote: Ray, We've got a paper due at the end of March, and 15-20K hours would let us finish that up on time. thanks, chris On 3/7/2017 3:00 PM, Bair, Raymond A. wrote: > Dear Chris, > > April starts the 2nd half of FY17, and you will be getting the second half allocation then. How much more time do you need between now and March 31? > > Regards, > > Ray > > > ----------------------------------------- > Ray Bair > Argonne National Laboratory > and the University of Chicago > > > On 3/3/17, 11:43 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: > > Hello, > > A change in allocation has been requested: > > Requester: cpowell (Chris Powell) > Project: XRayFuelSpray > Title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics > Description: The cavitation and diesel spray modeling sub-tasks are being performed using the OpenFOAM framework. These tasks will consume the bulk of the requested core hours. A number of state-of-the-art numerical solvers have been implemented in OpenFOAM; namely incompressible-liquid and fully compressible homogeneous relaxation cavitation models and more recently a fully compressible homogeneous relaxation cavitation model which includes non-condensable gas modeling (HRMFoam). This code can handle both cavitating as well as flash boiling flows. These codes were developed by Schmidt et al at the University of Massachusetts-Amherst, and we continue to collaborate with them on code development and efficiency improvements. The HRMFoam solver has been tested for scalability and shown good performance on up to 300 processors. We typically run jobs with 64 to 128 processors. > > The diesel spray sub-task in OpenFOAM studies turbulent, aerodynamic, and hybrid spray atomization sub-model formulations for high-pressure fuel sprays via customized sub-models. The target sprays for simulation will be the Engine Combustion Network (ECN) non-vaporizing Spray A and Spray D conditions. LCRC resources will enable GA Tech to rapidly assess model formulations and sensitivities over a wide range of liquid Weber and Reynolds number conditions, probing the relevance of turbulence and aerodynamic atomization mechanisms at limiting conditions. > > The combusting spray sub-task in OpenFOAM investigates the coupling of spray sub-models and turbulence-chemistry interactions for spray combustion through the development and implementation of a Large-Eddy Simulation (LES) framework. The target sprays for simulation will be the Engine Combustion Network (ECN) reacting Spray A condition. LCRC resources facilitate the simulation of multiple LES realizations in order to obtain reliable statistics for predicted spray and combustion parameters, which is essential for comparison against experimental measurements from APS and GA Tech. These comparisons will enable an in-depth study of both the instantaneous and average coupling between spray-induced turbulence structures and the resultant chemistry and flame evolution. > > The tasks described above will be run on 64 to 128 processors and will consume the bulk of the allocation requirement. Due to the conclusion of several sub-tasks from the previous FY, our allocation requirement for FY17 is reduced. > Tomographic reconstruction of APS data will be conducted using the TomoPy software package developed at Argonne. We run single-node jobs to efficiently perform multiple reconstructions of large (hundred-GB) datasets; these typically require less than 100 core hours each, but have high RAM requirements. This task will benefit substantially from the availability of the more powerful Haswell nodes and we are investigating the possibility of purchasing dedicated nodes on the new Bebop machine for this purpose. > > We have also successfully compiled a Paraview server on Blues which allows us to rapidly visualize the results of our simulations and reconstructions without having to wait to download the data to our local computer. This allows us to achieve rapid turnaround of results during beamtime. This is a significant aid to the experimental program. The server runs typically on a single Blues ‘biggpu’ node (16 cores) which allows us to open simulation datasets that are much larger than the RAM available on our desktop computers. This task requires relatively few core hours, since the server is only used for a few hours at a time. > Current: undetermined amount > Justification: Due to the conclusion of several sub-tasks, and plans to purchase dedicated nodes, our allocation requirements for FY17 are much less than in the previous year, and will not exceed 0.5M core hours. > > Requested: 20000 > > A specific reason has been given: > In our collaboration with Georgia Tech, we are working on fuel injection simulations for an upcoming publication. Additional CFD runs were required for grid convergence, and more CPU hours are required to complete this work. > > This needs to be approved and the final allocation amount decided upon. > > Thank You, > The LCRC Accounts System > > -- Christopher F. Powell, Ph.D. Center for Transportation Research Argonne National Laboratory 9700 South Cass Avenue Bldg 362, Room C217 Argonne, IL 60439 email: powell(a)anl.gov phone: (630) 252-9027 fax: (630) 252-3443

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 06 Mar '17

06 Mar '17

Hello, A change in allocation has been requested: Requester: heinonen (Olle Heinonen) Project: Meso Title: Mesoscale computational materials science Description: This project has three thrusts: (i) development of computational codes and algorithms for mesoscale modeling of inhomogeneous materials systems and applications of these codes, (ii) modeling of magnetic micron- and nano-scale systems, and (iii) first-principle modeling of metal/oxide/metal heterostructures and complex oxides. In the first thrust, we are developing scalable finite element codes for coupled elastic-ferroic systems, as well as charged particles immersed in a dielectric fluid subjected to electric fields. The code developments build on libraries such as libmesh and PETSc to achieve portable, scalable and highly efficiently parallelized codes. In particular, we are developing codes that combine fast-multipole methods with boundary element methods (FMM-BEM)to calculate long-range electrostatic and magnetostatic fields from bound and free surface charges. Within the MOOSE framework, we are also developing solvers for coupled ferroelectric/ferromagnetic-elastic systems. We will in the coming year integrate the FMM-BEM method with the MOOSE framework and apply the codes to study ferroelectric or ferromagnetic thin films and nanoparticles, as well as inhomogeneous composite multiferroic systems. We are in contact with a number of experimental groups with which we expect to collaborate. The work in this first thrust is integrated into codes under the MICCoM computational materials center led by Giulia Galli. The codes developed under this thrust are also massively scalable to leadership-class computer platforms. The programming language for these developments is C++. As we move from development to production phases we expect to run on up to 248 cores on fusion and blues. The MOOSE framework has been shown to scale almost linearly up to some 100k cores on BG/Q without any BG/Q-specific optimization. For the FMM-BEM method we have already shown linear scaling with the number of cores (using hybrid MPI-threading) to hundreds of cores. In the second thrust we are modeling static and dynamic magnet configurations in a variety of magnetic systems. This work is in collaboration with a number of experimental groups at Argonne and around the world. The workhorse of this thrust is a homemade micromagnetic code, or set of codes, in FORTRAN90 parallellized using OpenMP in addition to mkl libraries (in particular mkl dft); they only run on one node at a time but have been optimized and are highly efficient, typically running at 95% efficiency on eight to 32 cores. These runs typically consume one node on fusion or blues (all cores on the node), including the msd queue's Haswell nodes for one to 30 wall-hours. In thrust (iii), we are using the SIESTA/smeagol (a development version), quantum espresso (QE) and FHI-aims density functional theory based codes. All have been built and optimized for blues/fusion and we can routinely run on up to a few hundred cores (up to some tens of thousand cores on BG/Q) using hybrid parallelization (smeagol) or just MPI tasks (QE and FHI-aims). We are continuing to model electronic structure and transport properties of oxide heterostructures and complex oxides. The focus is shifting a bit FY2017 since this thrust will be aligned with the BES-funded computational materials center (led by ORNL). We expect there to be about seven members in this project (PI, two co-PIs, three post-docs and one graduate student). Current: undetermined amount Justification: The codes we will use have all been tested and optimized and run efficiently on blues. Requested: 524503 A specific reason has been given: We have run out of allocation. The project is servicing three different BES-funded projects (MICCoM, condensed matter theory FWP in MSD, and the computational materials science center led by Paul Kent at ORNL but with a large presence at ANL), and also the CHiMaD center of which I am a PI. We need another 500k core hours for FY2017 - we try to run as much as possible on the msd queue but it only has 30 nodes and we have to run on blues and the haswell queue, too. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 03 Mar '17

03 Mar '17

Hello, A change in allocation has been requested: Requester: cpowell (Chris Powell) Project: XRayFuelSpray Title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics Description: The cavitation and diesel spray modeling sub-tasks are being performed using the OpenFOAM framework. These tasks will consume the bulk of the requested core hours. A number of state-of-the-art numerical solvers have been implemented in OpenFOAM; namely incompressible-liquid and fully compressible homogeneous relaxation cavitation models and more recently a fully compressible homogeneous relaxation cavitation model which includes non-condensable gas modeling (HRMFoam). This code can handle both cavitating as well as flash boiling flows. These codes were developed by Schmidt et al at the University of Massachusetts-Amherst, and we continue to collaborate with them on code development and efficiency improvements. The HRMFoam solver has been tested for scalability and shown good performance on up to 300 processors. We typically run jobs with 64 to 128 processors. The diesel spray sub-task in OpenFOAM studies turbulent, aerodynamic, and hybrid spray atomization sub-model formulations for high-pressure fuel sprays via customized sub-models. The target sprays for simulation will be the Engine Combustion Network (ECN) non-vaporizing Spray A and Spray D conditions. LCRC resources will enable GA Tech to rapidly assess model formulations and sensitivities over a wide range of liquid Weber and Reynolds number conditions, probing the relevance of turbulence and aerodynamic atomization mechanisms at limiting conditions. The combusting spray sub-task in OpenFOAM investigates the coupling of spray sub-models and turbulence-chemistry interactions for spray combustion through the development and implementation of a Large-Eddy Simulation (LES) framework. The target sprays for simulation will be the Engine Combustion Network (ECN) reacting Spray A condition. LCRC resources facilitate the simulation of multiple LES realizations in order to obtain reliable statistics for predicted spray and combustion parameters, which is essential for comparison against experimental measurements from APS and GA Tech. These comparisons will enable an in-depth study of both the instantaneous and average coupling between spray-induced turbulence structures and the resultant chemistry and flame evolution. The tasks described above will be run on 64 to 128 processors and will consume the bulk of the allocation requirement. Due to the conclusion of several sub-tasks from the previous FY, our allocation requirement for FY17 is reduced. Tomographic reconstruction of APS data will be conducted using the TomoPy software package developed at Argonne. We run single-node jobs to efficiently perform multiple reconstructions of large (hundred-GB) datasets; these typically require less than 100 core hours each, but have high RAM requirements. This task will benefit substantially from the availability of the more powerful Haswell nodes and we are investigating the possibility of purchasing dedicated nodes on the new Bebop machine for this purpose. We have also successfully compiled a Paraview server on Blues which allows us to rapidly visualize the results of our simulations and reconstructions without having to wait to download the data to our local computer. This allows us to achieve rapid turnaround of results during beamtime. This is a significant aid to the experimental program. The server runs typically on a single Blues ‘biggpu’ node (16 cores) which allows us to open simulation datasets that are much larger than the RAM available on our desktop computers. This task requires relatively few core hours, since the server is only used for a few hours at a time. Current: undetermined amount Justification: Due to the conclusion of several sub-tasks, and plans to purchase dedicated nodes, our allocation requirements for FY17 are much less than in the previous year, and will not exceed 0.5M core hours. Requested: 20000 A specific reason has been given: In our collaboration with Georgia Tech, we are working on fuel injection simulations for an upcoming publication. Additional CFD runs were required for grid convergence, and more CPU hours are required to complete this work. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 03 Mar '17

03 Mar '17

Hello, A change in allocation has been requested: Requester: linyun (Linyun Liang) Project: Meso_Nuclear Title: Gas bubble kinetics in an irradiated U-Mo using a mesoscale simulation approach Description: A mesoscale simulation approach is developed to study the gas bubble evolution kinetics in irradiated U-Mo alloy fuels. Density functional theory is first utilized to predict the defect formation energies, the Xe-U-Mo interfacial energy, and the elastic constants of U-Mo alloys, among other quantities. These obtained material properties are then incorporated into a mesoscale model to study the kinetics of gas bubbles in U-Mo under irradiation. By developing a new free energy model of Xe gas bubble based on the van der Walls equation of state, the gas bubble pressure can be estimated. The effects of vacancy and self-interstitial atom (SIA) diffusion on the gas bubble pressure will be further investigated and validated. The events employed in the simulation are the absorption and emission of a vacancy, a SIA, and an interstitial U by a single Xe bubble, while an event associated with SIA loops or substitutional U is neglected here for simplicity. We will systemat ically investigate the effect of Xe, vacancy, and SIA concentration, fission defect generation and their recombination, sink strength, grain boundaries, and elastic interaction on the growth kinetics of gas bubble. The swelling of U-Mo due to fission gas bubble will be simulated and compared to experimental observations. Our study aims to provide insight into understanding of the structural damage in irradiated U-Mo materials. In our phase field model, we have four highly non-linear, coupled partial differential equations which must be solved numerically. In the meaning time, the elastic equilibrium equation is also included in each time step, which requires to obtain the elastic solution at the equilibrium state for each time step and can significantly increase the cost of the simulation time. And also in order to avoid the partial loss of interfacial energy, we will extend our model to three dimensional system. Therefore, to effectively solve this problem, we will need about 150,000 core hours including the 6,000 core-hour to debug and test the algorithm and code and 144,000 core-hours (4 cores/node * 4 nodes * 10 hours of run time * 800 different runs) for the nucleation and growth of gas bubbles calculations. We will use our developed FFTW code to solve the problem. By using the Fusion and Blues computing resources will greatly accelerated the code development and calculations. In conclusion, the total core-hours requested for this project throughout the FY2016 will be 150,000 core-hours. Fig. 1 Simulated results of gas bubbles nucleation and growth under the irradiated U-7Mo. Current: undetermined amount Justification: Requested: 300000 A specific reason has been given: I have run out of my core-hours for this project. I request to have additional core-hours from now. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Roberts, John E. 01 Mar '17

01 Mar '17

Time granted. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 2/28/17, 11:37 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: vama (Alvaro VazquezMayagoitia) Project: TDSE-APS Title: Time dependend quantum approaches to support APS X-ray experiments Description: We plan to run ~80 calculations using 1 node each for ~1-10 hrs. We recently added OpenMP to our code and applied vectorization techniques which reduced 10x times the walltime of our calculations. we plan to extend our parallelization with MPI. Most of the calculations will be carried out by two summer interns. Current: undetermined amount Justification: We currently run a binary that computes transitions states between core electrons to continuum states and internal transitions. We recently modified our serial code to use multiple threads. Our code scales to 32 cores per node. We plan to apply MPI parallelization. Requested: 50000 A specific reason has been given: We would like to continue using LCRC resources for our research in x-ray pulses in atoms and molecules. Blues has been extremely useful, and we recently published our results in a peer-reviewed PRA paper: Normal Auger processes with ultrashort x-ray pulses in neon Raymond Sullivan, Junteng Jia, Álvaro Vázquez-Mayagoitia, and Antonio Picón Phys. Rev. A 94, 043421 – Published 26 October 2016 https://journals.aps.org/pra/abstract/10.1103/PhysRevA.94.043421#fulltext Thanks Alvaro Vazquez-Mayagoitia This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0