June 2017 - Allocations-admins - mailman3.cels.anl.gov

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 27 Jun '17

27 Jun '17

This request looks OK to me. In the past, FOAM has only requested time when they needed it, and they have a reasonable justification. Recommend we approve. Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 6/26/17, 11:16 PM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: jacob (Robert Jacob) Project: foam Title: Global Climate Modeling with the Fast Ocean Atmosphere Model Description: Our major objectives for FY15 focus on the ensemble based parameter estimation and coupled data assimilation in a coupled ocean-atmosphere general circulation model (CGCM). We will use FOAM as the CGCM. The uncertainties of model parameters are one of the important sources of model bias in a CGCM. The traditional method of tuning parameters through trial-and-error sensitivity experiments is subjective and very inefficient. An ensemble-based filter (EnKF) potentially provides a novel approach for systematic parameter estimation in a CGCM. In FY14 we have successful implemented ensemble based parameter estimation in FOAM for both single parameter estimation and multiple parameter in a twin model framework. In FY15 we will continue this study. We will investigate the ensemble based parameter estimation using real observations. We have successfully developed a coupled data assimilation system in FOAM using EnKF in FY13 and FY14. However the coupled data assimilation system is far from mature. In FY15 we will continue to work on the improvement of the system. We mainly focus on improving the analysis quality by applying coupled covariance to ocean. We also will investigate the challenges of coupled data assimilation using real observations and test some possible solutions. In FY15, We will further investigate the coupled data assimilation and parameter estimation in FOAM this year. We will investigate single and multi-parameter estimations in FOAM using real observations. We will use model short time forecast and long-term variabilities of model simulation to valid the estimations Good parameter estimation requires good data assimilation. We also need to refine our pure data assimilation skills that will provide support for us to estimate parameter using real observation in the future. We will investigate the effect of coupled covariance to the quality of analysis data in both perfect framework and real world system. For the parameter estimation experiments, we request fifty members (50 copies) will be used for each experiment. We expect about 150 different sets of experiments for parameter estimation. We measured the cost of 1 core-hours per simulated year. Each simulation should take 30 years such that the total model years are: 1. Parameter estimation experiments: 50x30x150 = 225k hours For the validation of parameter estimation, we request 200 10-year forecast experiments and 30 100-year assimilations. The total model years for the experiments are: 2. Parameter estimation validation 50*(100*30+200*10)=250k hours For the improvement of coupled data assimilation and the testing in both perfect framework and real world system we also request 150 different sets of experiments. Each experiment uses fifty members and 30 model years. 3. Pure data assimilation experiments for improving the system 50x30x150 = 225k hours Total core hours for experiments: 225+225+250=700k core hours Current: undetermined amount Justification: The total experiments are independent and can be run simultaneously. FOAM is very scalable between 4 and 64 processors. We choose to run an individual ensemble member on 16 or 32 because the additional parallelism comes from the ensemble members. We could effectively use the entire machine if it was available. Requested: 100000 A specific reason has been given: We will be running some experiments with the data-assimilation version of FOAM to perpare a technical paper on its design. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 26 Jun '17

26 Jun '17

Hello, A change in allocation has been requested: Requester: jacob (Robert Jacob) Project: foam Title: Global Climate Modeling with the Fast Ocean Atmosphere Model Description: Our major objectives for FY15 focus on the ensemble based parameter estimation and coupled data assimilation in a coupled ocean-atmosphere general circulation model (CGCM). We will use FOAM as the CGCM. The uncertainties of model parameters are one of the important sources of model bias in a CGCM. The traditional method of tuning parameters through trial-and-error sensitivity experiments is subjective and very inefficient. An ensemble-based filter (EnKF) potentially provides a novel approach for systematic parameter estimation in a CGCM. In FY14 we have successful implemented ensemble based parameter estimation in FOAM for both single parameter estimation and multiple parameter in a twin model framework. In FY15 we will continue this study. We will investigate the ensemble based parameter estimation using real observations. We have successfully developed a coupled data assimilation system in FOAM using EnKF in FY13 and FY14. However the coupled data assimilation system is far from mature. In FY15 we will continue to work on the improvement of the system. We mainly focus on improving the analysis quality by applying coupled covariance to ocean. We also will investigate the challenges of coupled data assimilation using real observations and test some possible solutions. In FY15, We will further investigate the coupled data assimilation and parameter estimation in FOAM this year. We will investigate single and multi-parameter estimations in FOAM using real observations. We will use model short time forecast and long-term variabilities of model simulation to valid the estimations Good parameter estimation requires good data assimilation. We also need to refine our pure data assimilation skills that will provide support for us to estimate parameter using real observation in the future. We will investigate the effect of coupled covariance to the quality of analysis data in both perfect framework and real world system. For the parameter estimation experiments, we request fifty members (50 copies) will be used for each experiment. We expect about 150 different sets of experiments for parameter estimation. We measured the cost of 1 core-hours per simulated year. Each simulation should take 30 years such that the total model years are: 1. Parameter estimation experiments: 50x30x150 = 225k hours For the validation of parameter estimation, we request 200 10-year forecast experiments and 30 100-year assimilations. The total model years for the experiments are: 2. Parameter estimation validation 50*(100*30+200*10)=250k hours For the improvement of coupled data assimilation and the testing in both perfect framework and real world system we also request 150 different sets of experiments. Each experiment uses fifty members and 30 model years. 3. Pure data assimilation experiments for improving the system 50x30x150 = 225k hours Total core hours for experiments: 225+225+250=700k core hours Current: undetermined amount Justification: The total experiments are independent and can be run simultaneously. FOAM is very scalable between 4 and 64 processors. We choose to run an individual ensemble member on 16 or 32 because the additional parallelism comes from the ensemble members. We could effectively use the entire machine if it was available. Requested: 100000 A specific reason has been given: We will be running some experiments with the data-assimilation version of FOAM to perpare a technical paper on its design. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 16 Jun '17

16 Jun '17

Hello, A change in allocation has been requested: Requester: roberts (John Roberts) Project: support Title: Systems administration tasks on the LCRC Cluster(s). Description: NA Current: undetermined amount Justification: Requested: 1000 A specific reason has been given: Please ignore, this is a test after modifying the page code. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Roberts, John E. 14 Jun '17

14 Jun '17

This time has been granted. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 6/14/17, 9:43 AM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: jacob (Robert Jacob) Project: ACME Title: Accelerated Climate Modeling for Energy Description: ACME started with a branch of the Community Earth System Model (CESM). CESM is a fully-coupled, global climate model that provides state-of-the-art computer simulations of the Earth's past, present, and future climate states. ACME's atmosphere component is the Community Atmosphere Model. The CAM model has flexible formulations for atmospheric dynamics and has recently transitioned to the spectral finite element method coupled to an extensive suite of sub-grid physical parameterizations in its standard configuration. It runs on unstructured quadrilateral grids. The land model, CLM, contains a suite of column process parameterizations running at each grid point with no communication between grid points. CLM represents surface and subsurface water, energy, carbon, and nitrogen dynamics, with a nested hierarchical sub-grid parameterization that allows glaciers, lakes, urban areas, agricultural fields, managed forest, and natural (unmanaged) vegetation types to share space on each grid-cell. Incident radiation is intercepted in a two-layer canopy, with vegetation, soil, snow aging, and black carbon impacts on albedo. Subsurface processes include vertically resolved biogeochemistry, options for carbon and nutrient cycle parameterization, and recently improved treatment of wetlands and permafrost dynamics. In the fully-coupled configuration, the sea-ice and ocean (MPAS-CICE and MPAS-O) component models run with a nominal grid spacing of 1/10° (approximately 15km at the equator and 5km in polar regions) and, for MPAS-O, 100 levels in the vertical. CAM grids have both uniform and variable resolution configurations. The uniform grid with an average grid point spacing at the equator of 1/4° contains 86,400 elements, each with 30 vertical levels. Doubling the resolution to an average grid spacing of 1/8° requires 345,600 elements. The variable resolution configuration allows a small region of 1/8° grid spacing transitioning to an average global resolution of 1°. The CLM is run on the same grid as CAM. The output of the ACME consists of monthly means of several hundred quantities, plus daily averages of a subset of these quantities and hourly output of some key variabl es. ACME requires Fortran90 and C compilers and the MPI library to run. We will be using time on Blues to perform regular testing of ACME. ACME contains a large set of system tests that perform runs of the full model to test various functions. The runs sizes range from a few nodes to a few dozen nodes. The output sizes are relatively small and we will have modest storage needs. In FY17 we will continue regular testing with the gnu compiler and the large "acme_integration" test suite. In Current: undetermined amount Justification: Requested: 100000 A specific reason has been given: Our tests are running more frequently and using more time. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 14 Jun '17

14 Jun '17

Hello, A change in allocation has been requested: Requester: jacob (Robert Jacob) Project: ACME Title: Accelerated Climate Modeling for Energy Description: ACME started with a branch of the Community Earth System Model (CESM). CESM is a fully-coupled, global climate model that provides state-of-the-art computer simulations of the Earth's past, present, and future climate states. ACME's atmosphere component is the Community Atmosphere Model. The CAM model has flexible formulations for atmospheric dynamics and has recently transitioned to the spectral finite element method coupled to an extensive suite of sub-grid physical parameterizations in its standard configuration. It runs on unstructured quadrilateral grids. The land model, CLM, contains a suite of column process parameterizations running at each grid point with no communication between grid points. CLM represents surface and subsurface water, energy, carbon, and nitrogen dynamics, with a nested hierarchical sub-grid parameterization that allows glaciers, lakes, urban areas, agricultural fields, managed forest, and natural (unmanaged) vegetation types to share space on each grid-cell. Incident radiation is intercepted in a two-layer canopy, with vegetation, soil, snow aging, and black carbon impacts on albedo. Subsurface processes include vertically resolved biogeochemistry, options for carbon and nutrient cycle parameterization, and recently improved treatment of wetlands and permafrost dynamics. In the fully-coupled configuration, the sea-ice and ocean (MPAS-CICE and MPAS-O) component models run with a nominal grid spacing of 1/10° (approximately 15km at the equator and 5km in polar regions) and, for MPAS-O, 100 levels in the vertical. CAM grids have both uniform and variable resolution configurations. The uniform grid with an average grid point spacing at the equator of 1/4° contains 86,400 elements, each with 30 vertical levels. Doubling the resolution to an average grid spacing of 1/8° requires 345,600 elements. The variable resolution configuration allows a small region of 1/8° grid spacing transitioning to an average global resolution of 1°. The CLM is run on the same grid as CAM. The output of the ACME consists of monthly means of several hundred quantities, plus daily averages of a subset of these quantities and hourly output of some key variabl es. ACME requires Fortran90 and C compilers and the MPI library to run. We will be using time on Blues to perform regular testing of ACME. ACME contains a large set of system tests that perform runs of the full model to test various functions. The runs sizes range from a few nodes to a few dozen nodes. The output sizes are relatively small and we will have modest storage needs. In FY17 we will continue regular testing with the gnu compiler and the large "acme_integration" test suite. In Current: undetermined amount Justification: Requested: 100000 A specific reason has been given: Our tests are running more frequently and using more time. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - Lattice-QCD
by Roberts, John E. 13 Jun '17

13 Jun '17

This time has been granted. Thanks. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 6/11/17, 12:40 PM, "allocations-admins on behalf of Bair, Raymond A." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of rbair(a)anl.gov> wrote: Let’s add 100K to Lattice-QCD too for Q3 ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 6/10/17, 12:03 PM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: sinclair (Donald Sinclair) Project: Lattice-QCD Title: Lattice simulations of QCD at finite baryon number density Description: A field theory in the Euclidean time regime can be quantized in terms of a partition function which is the functional integral over the c-number fields of the exponential of minus the action. In the Langevin equation, the fields evolve in a fictitious time such that the derivative of each field with respect to this time is minus the derivative of the action with respect to that field -- the drift term -- plus a gaussian distributed random number appropriately normalized. It can be shown that in the large 'time' limit the fields are distributed with weight exp(-S), where S is the action. The Langevin equation can be extended to the case where S is complex, by replacing the real fields by complex fields. Here it can only be demonstrated that the time averages of observables give the correct expectation values, when the region traversed by the fields is compact and S is holomorphic in the fields. When this Complex Langevin Equation (CLE) is applied to Lattice QCD, the gauge fields must be extended from SU(3) to SL(3,C) and the action involves the gauge fields and their inverses, but not their hermitian conjugates. The fermions are integrated out producing a power of the determinant of the Dirac operator. We obtain an effective action for the gauge fields by using the identity, determinant(M)^q=exp{q*ln[determinant(M)]} =exp{q*Trace[ln(M)]}. The trace is replaced by a stochastic estimator. The derivative with respect to the fields introduces an inverse of the Dirac operator M, which is evaluated using the conjugate gradient method. Unfortunately, since the determinant has zeros, the inverse of M and thus the drift term have poles, so that the drift term is not holomorphic but meromorphic in the fields. It is for this reason that, although the domain over which the fields evolve appears compact, convergence of the observables to their correct values is not guaranteed, and must be checked. We integrate the CLE numerically, inverting the Dirac operator using the conjugate gradient method. The fermion fields are placed on the sites of a 4-dimensional hypercubic lattice and the gauge fields on the links, making the action gauge invariant. If we label the sites by the 4-vector (x,y,z,t), we parallelize using MPI giving each task 1 t coordinate and nz adjacent z coordinates. The x and y coordinates are internal to each task. Each task then performs the same number of operations, and the communications are all nearest neighbour and homogeneous. In particular M only connects nearest-neighbour sites and is sparse. M is discretized using the staggered fermion method. Gauge cooling is implemented to minimize the sum over links of Tr{UU^dagger+(UU^dagger)^[-1]-2}, where U are the lattice gauge fields. Because we use 64-bit floating point (our previous simulations used 32-bit floating point except places where precision is critical), scaling is not as good as is in our previous work. In addition, since we can anticipate algorithmic changes, we have not optimized our conjugate gradient routine as aggressively as before. We plan to do this, as soon as we are satisfied that we have an optimum version. We only have absolute (Gflop) performances for 12^4 lattices measured on NERSC computers. The best such performance was 3.0 Gflops/core on Cori. We assume that the best performances on NERSC machines for 16^4 lattices would be similar. On 64 cores the performance of the code on a 16^4 lattice on the 32 core Haswell nodes of Blues was approximately 10% slower than on Cori. On the 16 core Blues nodes, the performance was approximately 25% slower than on Cori. For scaling studies on Blues, see the Large Allocation Efficiency section. We have performed runs on a 12^4 lattice at beta=6/g^2=5.6 (g is the lattice QCD bare coupling) and quark mass m=0.025, over a wide range of mu values. The results of these runs are in qualitative agreement with what is expected. However, there are quantitative deviations from our expectations. Running with the same beta and m values on a 16^4 lattice for a selection of mu values showed that these deviations are not a finite volume effect. On the 16^4 lattice we are able to simulate at weaker coupling (beta=5.7). At mu=0 we find far better agreement with known results than for beta=5.6. We are extending our beta=5.7 simulations to mu =/= 0 on Blues and Bridges at PSC to determine if there is similar improvement away from mu=0, and propose to continue these runs in FY2017. Note that we have less than 500,000 hours on Bridges until January, 2017. The current runs being performed on Blues are on 16^4 lattices, where we are using 64 cores (2 32-core Haswell nodes). Each job takes 50-60 wallclock-hours. We plan to run similar jobs in FY2017. We intend to run 2 such jobs concurrently, or 1 128 core job. When we are convinced that we have the code we wish to run in extensive production, we will apply optimizations that have proved effective with other codes, to the conjugate gradient routine where the jobs spend most of their time. Project website:www.hep.anl.gov/dks Current: undetermined amount Justification: We estimate that our code achieves approximately 3 Gflops per core on the Haswell nodes for the 64 cores we plan to use for production. 64 cores is the 'sweet spot' for 16^4 lattices where the per core performance is maximum. The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 32-core Haswell nodes on Blues. Each run used all 32 cores on each node, and 1 task/core. 32 cores haswell = 51m44.125s 64 cores haswell = 21m41.861s 128 cores haswell = 13m3.596s 256 cores haswell = 9m43.317s The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 16-core nodes on Blues. Each run used all 16 cores on each node, and 1 task/core. 16 cores = 129m35.228s 32 cores = 52m37.305s 64 cores = 25m35.144s 128 cores = 17m55.801s 256 cores = 16m40.058s As indicated above these numbers would be expected to improve when we apply further optimizations to the conjugate gradient code, which have proved effective on other, similar codes. Our request is based on performing 2 64-core or 1 128-core job continuously, assuming that Blues will be available for production 90% of the time. Requested: 100000 A specific reason has been given: Needed to complete present runs This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - ElectrolyteRedoxFlow
by Roberts, John E. 13 Jun '17

13 Jun '17

This time has been granted. Thanks. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 6/11/17, 12:39 PM, "allocations-admins on behalf of Bair, Raymond A." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of rbair(a)anl.gov> wrote: Let’s go with adding 100K for Q3 for ElectrolyteRedoxFlow ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 6/9/17, 1:28 PM, "Cheng, Lei" <leicheng(a)anl.gov> wrote: Dear Ray, Since there are only 21 days left this quarter, I think 100000 should be sufficient to get us going for three weeks. Thank you so much for considering additional allocation for me! Lei -----Original Message----- From: Bair, Raymond A. Sent: Friday, June 09, 2017 12:25 PM To: Cheng, Lei <leicheng(a)anl.gov> Cc: Bair, Raymond A. <rbair(a)anl.gov> Subject: Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - ElectrolyteRedoxFlow Dear Lei Cheng, Changes to project allocations apply to the current quarter (April 1 – June 30). So, there are 21 days left in the quarter. How much time is needed for that period (factoring in that many others are running too)? Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 6/9/17, 10:48 AM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: leicheng (Lei Cheng) Project: ElectrolyteRedoxFlow Title: discoveries of electrolyte for redox flow batteries Description: In this renewal work, we will focus on stability studies of redox-active molecules for aqueous redox flow battery application. A short list of redox-active molecules for NRF application down-selected through high-throughput screening scheme were synthesized and tested experimentally (Moore and Brushett). The validation work shows that most of the compounds have irreversible redox activity, suggesting their instability in charged states. Similar phenomenon has been observed for molecules screened for aqueous redox flow (ARF) application. For example, catechol molecule can potentially be used as active catholyte materials in ARF battery. The electron-withdrawing group nitro is added to increase the oxidation potential of the catechol base molecule and it was predicted to increase the oxidation potential by 0.2 V. However, the experimental cyclic voltammetry measurements show that the oxidation of nitro-substituted catechol is not reversible (no reduction observed ) though the oxidation potentials agree well with calculation (courtesy of Zhang and Su, ANL). We are investigating addition reactions (1,2-addition and 1,4-addition) by H2O molecule as possible degradation reaction mechanism on a series of hydroquinone compounds with available experimental reversibility data. This study will shed light on possible stability predictors such as substitution site and electron-activating strength of substituent, as well as designing more stable molecules for this category of compounds. Current: undetermined amount Justification: All calculations will be performed using Gaussian. This package is considered very efficient and reliable for quantum chemistry calculations. Gaussian is not very scalable but it provides really robust geometry optimizer and the G3/G4 methods are built-in. The size of molecules we propose to study is very typical for the computational software (Gaussian) so they scale reasonably well for most DFT calculation. The higher level calculations will consume more memory as well as computational time. Requested: 350000 A specific reason has been given: I had to do some transition state search to look for decomposition pathways of some molecules using Gaussian and NEB within VASP which used up a lot of CPU time. Will need more computing time to continue these TS searches and continue the project. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - Lattice-QCD
by Bair, Raymond A. 11 Jun '17

11 Jun '17

Let’s add 100K to Lattice-QCD too for Q3 ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 6/10/17, 12:03 PM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: sinclair (Donald Sinclair) Project: Lattice-QCD Title: Lattice simulations of QCD at finite baryon number density Description: A field theory in the Euclidean time regime can be quantized in terms of a partition function which is the functional integral over the c-number fields of the exponential of minus the action. In the Langevin equation, the fields evolve in a fictitious time such that the derivative of each field with respect to this time is minus the derivative of the action with respect to that field -- the drift term -- plus a gaussian distributed random number appropriately normalized. It can be shown that in the large 'time' limit the fields are distributed with weight exp(-S), where S is the action. The Langevin equation can be extended to the case where S is complex, by replacing the real fields by complex fields. Here it can only be demonstrated that the time averages of observables give the correct expectation values, when the region traversed by the fields is compact and S is holomorphic in the fields. When this Complex Langevin Equation (CLE) is applied to Lattice QCD, the gauge fields must be extended from SU(3) to SL(3,C) and the action involves the gauge fields and their inverses, but not their hermitian conjugates. The fermions are integrated out producing a power of the determinant of the Dirac operator. We obtain an effective action for the gauge fields by using the identity, determinant(M)^q=exp{q*ln[determinant(M)]} =exp{q*Trace[ln(M)]}. The trace is replaced by a stochastic estimator. The derivative with respect to the fields introduces an inverse of the Dirac operator M, which is evaluated using the conjugate gradient method. Unfortunately, since the determinant has zeros, the inverse of M and thus the drift term have poles, so that the drift term is not holomorphic but meromorphic in the fields. It is for this reason that, although the domain over which the fields evolve appears compact, convergence of the observables to their correct values is not guaranteed, and must be checked. We integrate the CLE numerically, inverting the Dirac operator using the conjugate gradient method. The fermion fields are placed on the sites of a 4-dimensional hypercubic lattice and the gauge fields on the links, making the action gauge invariant. If we label the sites by the 4-vector (x,y,z,t), we parallelize using MPI giving each task 1 t coordinate and nz adjacent z coordinates. The x and y coordinates are internal to each task. Each task then performs the same number of operations, and the communications are all nearest neighbour and homogeneous. In particular M only connects nearest-neighbour sites and is sparse. M is discretized using the staggered fermion method. Gauge cooling is implemented to minimize the sum over links of Tr{UU^dagger+(UU^dagger)^[-1]-2}, where U are the lattice gauge fields. Because we use 64-bit floating point (our previous simulations used 32-bit floating point except places where precision is critical), scaling is not as good as is in our previous work. In addition, since we can anticipate algorithmic changes, we have not optimized our conjugate gradient routine as aggressively as before. We plan to do this, as soon as we are satisfied that we have an optimum version. We only have absolute (Gflop) performances for 12^4 lattices measured on NERSC computers. The best such performance was 3.0 Gflops/core on Cori. We assume that the best performances on NERSC machines for 16^4 lattices would be similar. On 64 cores the performance of the code on a 16^4 lattice on the 32 core Haswell nodes of Blues was approximately 10% slower than on Cori. On the 16 core Blues nodes, the performance was approximately 25% slower than on Cori. For scaling studies on Blues, see the Large Allocation Efficiency section. We have performed runs on a 12^4 lattice at beta=6/g^2=5.6 (g is the lattice QCD bare coupling) and quark mass m=0.025, over a wide range of mu values. The results of these runs are in qualitative agreement with what is expected. However, there are quantitative deviations from our expectations. Running with the same beta and m values on a 16^4 lattice for a selection of mu values showed that these deviations are not a finite volume effect. On the 16^4 lattice we are able to simulate at weaker coupling (beta=5.7). At mu=0 we find far better agreement with known results than for beta=5.6. We are extending our beta=5.7 simulations to mu =/= 0 on Blues and Bridges at PSC to determine if there is similar improvement away from mu=0, and propose to continue these runs in FY2017. Note that we have less than 500,000 hours on Bridges until January, 2017. The current runs being performed on Blues are on 16^4 lattices, where we are using 64 cores (2 32-core Haswell nodes). Each job takes 50-60 wallclock-hours. We plan to run similar jobs in FY2017. We intend to run 2 such jobs concurrently, or 1 128 core job. When we are convinced that we have the code we wish to run in extensive production, we will apply optimizations that have proved effective with other codes, to the conjugate gradient routine where the jobs spend most of their time. Project website:www.hep.anl.gov/dks Current: undetermined amount Justification: We estimate that our code achieves approximately 3 Gflops per core on the Haswell nodes for the 64 cores we plan to use for production. 64 cores is the 'sweet spot' for 16^4 lattices where the per core performance is maximum. The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 32-core Haswell nodes on Blues. Each run used all 32 cores on each node, and 1 task/core. 32 cores haswell = 51m44.125s 64 cores haswell = 21m41.861s 128 cores haswell = 13m3.596s 256 cores haswell = 9m43.317s The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 16-core nodes on Blues. Each run used all 16 cores on each node, and 1 task/core. 16 cores = 129m35.228s 32 cores = 52m37.305s 64 cores = 25m35.144s 128 cores = 17m55.801s 256 cores = 16m40.058s As indicated above these numbers would be expected to improve when we apply further optimizations to the conjugate gradient code, which have proved effective on other, similar codes. Our request is based on performing 2 64-core or 1 128-core job continuously, assuming that Blues will be available for production 90% of the time. Requested: 100000 A specific reason has been given: Needed to complete present runs This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - ElectrolyteRedoxFlow
by Bair, Raymond A. 11 Jun '17

11 Jun '17

Let’s go with adding 100K for Q3 for ElectrolyteRedoxFlow ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 6/9/17, 1:28 PM, "Cheng, Lei" <leicheng(a)anl.gov> wrote: Dear Ray, Since there are only 21 days left this quarter, I think 100000 should be sufficient to get us going for three weeks. Thank you so much for considering additional allocation for me! Lei -----Original Message----- From: Bair, Raymond A. Sent: Friday, June 09, 2017 12:25 PM To: Cheng, Lei <leicheng(a)anl.gov> Cc: Bair, Raymond A. <rbair(a)anl.gov> Subject: Re: [allocations-admins] [LCRC Accounts] Project Allocation Request - ElectrolyteRedoxFlow Dear Lei Cheng, Changes to project allocations apply to the current quarter (April 1 – June 30). So, there are 21 days left in the quarter. How much time is needed for that period (factoring in that many others are running too)? Regards, Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 6/9/17, 10:48 AM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: leicheng (Lei Cheng) Project: ElectrolyteRedoxFlow Title: discoveries of electrolyte for redox flow batteries Description: In this renewal work, we will focus on stability studies of redox-active molecules for aqueous redox flow battery application. A short list of redox-active molecules for NRF application down-selected through high-throughput screening scheme were synthesized and tested experimentally (Moore and Brushett). The validation work shows that most of the compounds have irreversible redox activity, suggesting their instability in charged states. Similar phenomenon has been observed for molecules screened for aqueous redox flow (ARF) application. For example, catechol molecule can potentially be used as active catholyte materials in ARF battery. The electron-withdrawing group nitro is added to increase the oxidation potential of the catechol base molecule and it was predicted to increase the oxidation potential by 0.2 V. However, the experimental cyclic voltammetry measurements show that the oxidation of nitro-substituted catechol is not reversible (no reduction observed ) though the oxidation potentials agree well with calculation (courtesy of Zhang and Su, ANL). We are investigating addition reactions (1,2-addition and 1,4-addition) by H2O molecule as possible degradation reaction mechanism on a series of hydroquinone compounds with available experimental reversibility data. This study will shed light on possible stability predictors such as substitution site and electron-activating strength of substituent, as well as designing more stable molecules for this category of compounds. Current: undetermined amount Justification: All calculations will be performed using Gaussian. This package is considered very efficient and reliable for quantum chemistry calculations. Gaussian is not very scalable but it provides really robust geometry optimizer and the G3/G4 methods are built-in. The size of molecules we propose to study is very typical for the computational software (Gaussian) so they scale reasonably well for most DFT calculation. The higher level calculations will consume more memory as well as computational time. Requested: 350000 A specific reason has been given: I had to do some transition state search to look for decomposition pathways of some molecules using Gaussian and NEB within VASP which used up a lot of CPU time. Will need more computing time to continue these TS searches and continue the project. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 10 Jun '17

10 Jun '17

Hello, A change in allocation has been requested: Requester: sinclair (Donald Sinclair) Project: Lattice-QCD Title: Lattice simulations of QCD at finite baryon number density Description: A field theory in the Euclidean time regime can be quantized in terms of a partition function which is the functional integral over the c-number fields of the exponential of minus the action. In the Langevin equation, the fields evolve in a fictitious time such that the derivative of each field with respect to this time is minus the derivative of the action with respect to that field -- the drift term -- plus a gaussian distributed random number appropriately normalized. It can be shown that in the large 'time' limit the fields are distributed with weight exp(-S), where S is the action. The Langevin equation can be extended to the case where S is complex, by replacing the real fields by complex fields. Here it can only be demonstrated that the time averages of observables give the correct expectation values, when the region traversed by the fields is compact and S is holomorphic in the fields. When this Complex Langevin Equation (CLE) is applied to Lattice QCD, the gauge fields must be extended from SU(3) to SL(3,C) and the action involves the gauge fields and their inverses, but not their hermitian conjugates. The fermions are integrated out producing a power of the determinant of the Dirac operator. We obtain an effective action for the gauge fields by using the identity, determinant(M)^q=exp{q*ln[determinant(M)]} =exp{q*Trace[ln(M)]}. The trace is replaced by a stochastic estimator. The derivative with respect to the fields introduces an inverse of the Dirac operator M, which is evaluated using the conjugate gradient method. Unfortunately, since the determinant has zeros, the inverse of M and thus the drift term have poles, so that the drift term is not holomorphic but meromorphic in the fields. It is for this reason that, although the domain over which the fields evolve appears compact, convergence of the observables to their correct values is not guaranteed, and must be checked. We integrate the CLE numerically, inverting the Dirac operator using the conjugate gradient method. The fermion fields are placed on the sites of a 4-dimensional hypercubic lattice and the gauge fields on the links, making the action gauge invariant. If we label the sites by the 4-vector (x,y,z,t), we parallelize using MPI giving each task 1 t coordinate and nz adjacent z coordinates. The x and y coordinates are internal to each task. Each task then performs the same number of operations, and the communications are all nearest neighbour and homogeneous. In particular M only connects nearest-neighbour sites and is sparse. M is discretized using the staggered fermion method. Gauge cooling is implemented to minimize the sum over links of Tr{UU^dagger+(UU^dagger)^[-1]-2}, where U are the lattice gauge fields. Because we use 64-bit floating point (our previous simulations used 32-bit floating point except places where precision is critical), scaling is not as good as is in our previous work. In addition, since we can anticipate algorithmic changes, we have not optimized our conjugate gradient routine as aggressively as before. We plan to do this, as soon as we are satisfied that we have an optimum version. We only have absolute (Gflop) performances for 12^4 lattices measured on NERSC computers. The best such performance was 3.0 Gflops/core on Cori. We assume that the best performances on NERSC machines for 16^4 lattices would be similar. On 64 cores the performance of the code on a 16^4 lattice on the 32 core Haswell nodes of Blues was approximately 10% slower than on Cori. On the 16 core Blues nodes, the performance was approximately 25% slower than on Cori. For scaling studies on Blues, see the Large Allocation Efficiency section. We have performed runs on a 12^4 lattice at beta=6/g^2=5.6 (g is the lattice QCD bare coupling) and quark mass m=0.025, over a wide range of mu values. The results of these runs are in qualitative agreement with what is expected. However, there are quantitative deviations from our expectations. Running with the same beta and m values on a 16^4 lattice for a selection of mu values showed that these deviations are not a finite volume effect. On the 16^4 lattice we are able to simulate at weaker coupling (beta=5.7). At mu=0 we find far better agreement with known results than for beta=5.6. We are extending our beta=5.7 simulations to mu =/= 0 on Blues and Bridges at PSC to determine if there is similar improvement away from mu=0, and propose to continue these runs in FY2017. Note that we have less than 500,000 hours on Bridges until January, 2017. The current runs being performed on Blues are on 16^4 lattices, where we are using 64 cores (2 32-core Haswell nodes). Each job takes 50-60 wallclock-hours. We plan to run similar jobs in FY2017. We intend to run 2 such jobs concurrently, or 1 128 core job. When we are convinced that we have the code we wish to run in extensive production, we will apply optimizations that have proved effective with other codes, to the conjugate gradient routine where the jobs spend most of their time. Project website:www.hep.anl.gov/dks Current: undetermined amount Justification: We estimate that our code achieves approximately 3 Gflops per core on the Haswell nodes for the 64 cores we plan to use for production. 64 cores is the 'sweet spot' for 16^4 lattices where the per core performance is maximum. The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 32-core Haswell nodes on Blues. Each run used all 32 cores on each node, and 1 task/core. 32 cores haswell = 51m44.125s 64 cores haswell = 21m41.861s 128 cores haswell = 13m3.596s 256 cores haswell = 9m43.317s The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 16-core nodes on Blues. Each run used all 16 cores on each node, and 1 task/core. 16 cores = 129m35.228s 32 cores = 52m37.305s 64 cores = 25m35.144s 128 cores = 17m55.801s 256 cores = 16m40.058s As indicated above these numbers would be expected to improve when we apply further optimizations to the conjugate gradient code, which have proved effective on other, similar codes. Our request is based on performing 2 64-core or 1 128-core job continuously, assuming that Blues will be available for production 90% of the time. Requested: 100000 A specific reason has been given: Needed to complete present runs This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0