December 2015 - Allocations-admins - mailman3.cels.anl.gov

Re: [allocations-admins] Project Allocation Request for lipidmovement
by Bair, Raymond A. 21 Dec '15

21 Dec '15

Dear Ursula, Looking at the LCRC computer time usage for project lipidmovement, as of the last report 408K core-hours have been used out of 450K granted for the first half of the fiscal year (October 1, 2015 through March 31, 2016). Below, your new request for more time would add only 30K core hours. Is that correct? Regards, Ray ----------------------------------------- Ray Bair Computing, Environment, and Life Sciences Argonne National Laboratory and the University of Chicago TCS Building 240, Room 4122 9700 South Cass Avenue Argonne, IL 60439 email: rbair(at)anl.gov Phone: (630)252-5751 On 12/21/15, 2:20 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: >Hello, > >A change in allocation has been requested: > > Requester: perezsalas (Ursula Perez Salas) > Project: lipidmovement > Title: Curvature Effect on the Movement of Lipids Within and >Between Membranes > Description: Earlier, we have used a unilamellar lipid (DPPC) vesicle, >30 nm in diameter, solvated in water consisting of 10 6 coarse-grained >particles, to investigate what effect the curvature has on the free >energy of lipid flip-flop across the membrane. This size of vesicle is >similar to those used in neutron scattering exper- >iments to which we compare the results of the simulations. We found that >there is indeed a >small but noticeable curvature effect on the free energy of DPPC >flip-flop. > >Using vesicles similar to the DPPC model vesicle, we will build several >new vesicle systems >consisting either of POPS and POPC at two different cholesterol >concentrations and use these >systems to calculate the free energy for cholesterol flip-flop and >desorption to investigate the >effect of curvature on charged and uncharged membranes. Specifically, we >will investigate each >vesicle system at at three different temperatures for cholesterol >concentrations, leading to a >total of six systems. > >In order to estimate the rate of direct absorption of cholesterol, we >will run a series of small, flat >membrane simulations where we place a cholesterol molecule just outside >the membrane and let >it get absorbed without any external influence. Here we will run a set of >25 small simulations >measuring the absorption time per system, with two rate estimations per >simulation. > >To investigate the temperature dependence of the absorption rates we will >simulate one >POPS-cholesterol system and one POPC-cholesterol system at four different >temperatures, >yielding a total of 8 simulation sets. > >To calculate the needed free energy profiles we will use the umbrella >sampling technique which >is particularly useful when proper sampling can not be reached in a >simple long equilibrium >simulation. To do this, the system is constrained to sample a small >region of phase space with a >known external potential added to the system potential. The set of >constraints used to sample >the systems are independent of each other, which means that Umbrella >Sampling simulations >can be run in parallel. The output sampling data is combined using the >Weighted Histogram >Analysis method, post simulation. > > >**Estimate of number of core-hours: > > >Direct absorption simulations in the flat membrane systems: >25 × 8 = 200 simulations are needed. With each simulation running on one >node of 8 cores at 24 h/simulation (estimated from a benchmark of 0.0064 >days/ns) to get enough data, will yield a need for 200 × 24 × 8 = 38,400 >≈ 40,000 core-hours. > > >Vesicle system equilibration simulations: We need to equilibrate the >vesicle systems >before using them in the free energy calculations. A typical >equilibration time is 300 - 400 ns, >which with a benchmark time of 0.007 ns/day yields a need for 48 h per >simulation. Run on >512 cores, we would need 48 h/simulation × 6 simulation × 512 cores = >147456 ≈ 150,000 core-hours. > > >Cholesterol flip-flop and desorption free energy sampling in the >vesicles: 6 systems × >5 simulations/system = 30 sampling simulations are needed. With 30,000 >data points per sam- >pling region, collected at every 1 ps as a minimum data requirement and >with a benchmark time of 0.06 >days/ns, we need 32 h per simulation, but 35 - 40 h is desirable. A >vesicle simulation typically >600 cores to run at this pace, which yields 30 simulations × 40 >hours/simulation × 600 cores ≈ >720,000 core-hours. > >In total, we need 40, 000 + 150, 000 + 720, 000 ≈ 910,000 core-hours were >the 40,000 >core hours for the absorption estimate simulations are used on >independently run small simulations. > >We will use the molecular dynamics program NAMD, which is developed at >University of Illinois >at Urbana-Champaign and is freely available for download >(http://www.ks.uiuc.edu/Research/namd/) >NAMD is highly parallel and scales efficiently with the number of >particles in the system. >With the release of version 2.10 of NAMD we have seen an increase in >efficiency of about >15 % on other machines for the proposed vesicle systems. Thus we believe >that the >obtained core hours could be used more efficiently with the new version >of NAMD also on the >Blues and Fusion clusters. > > >Benchmarks obtained at the UIC Extreme cluster on a 1M coarse-grained >particle system: > >Cores | 80 | 160 | 240 | > >NAMD 2.9 | 76.9 ns/day | 108 ns/day | 120.7 ns/day | >NAMD 2.10 | 83.3 ns/day | 129.8 ns/day | 138.8 ns/day | > Current: undetermined amount >Justification: The vesicle systems have an obvious advantage of using the >clusters at Argonne, since they need hundreds of cores to reach peak >efficiency which is something that is far from feasible even on a high >end private desktop computer. > >Current benchmarks used to estimate the core-hours needed for the vesicle >system equilibration simulations are from old scaling tests. We have not >yet been able to try the new version of NAMD on the blues cluster, but >benchmarks obtained at the UIC Extreme cluster on a 1M coarse-grained >particle system shows a good scaling up to at least 240 cores and a >speedup of 10-20 % compared to NAMD 2.9: > >Cores | 80 | 160 | 240 | > >NAMD 2.9 | 76.9 ns/day | 108 ns/day | 120.7 ns/day | >NAMD 2.10 | 83.3 ns/day | 129.8 ns/day | 138.8 ns/day | > > >The obvious bottleneck in our project is the umbrella sampling of the >vesicle systems. The one order of magnitude slowdown that can be seen >when comparing the benchmark numbers between the equilibrium simulations >with the umbrella sampling simulations is due to the fact that the force >calculations needed for the external constraints applied to the systems >during sampling cannot be run in parallel. However, the code used to >apply the constraints is user made and we plan on revising e.g. the >update frequency of the lists needed to perform the force calculations >and the output frequency of our sampled variables. > > Requested: 30000 > >A specific reason has been given: >Our work has required the usage of all our current allocated simulation >time on half the time estimated. We are very close to having a >publication and the last of our simulations are critical for finishing >our analysis. Given our current usage and the simulations needed to >finish the analysis, we ask for 30 000 more core hours to be added to our >current allocation. > >This needs to be approved and the final allocation amount decided upon. > >Thank You, >The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 21 Dec '15

21 Dec '15

Hello, A change in allocation has been requested: Requester: perezsalas (Ursula Perez Salas) Project: lipidmovement Title: Curvature Effect on the Movement of Lipids Within and Between Membranes Description: Earlier, we have used a unilamellar lipid (DPPC) vesicle, 30 nm in diameter, solvated in water consisting of 10 6 coarse-grained particles, to investigate what effect the curvature has on the free energy of lipid flip-flop across the membrane. This size of vesicle is similar to those used in neutron scattering exper- iments to which we compare the results of the simulations. We found that there is indeed a small but noticeable curvature effect on the free energy of DPPC flip-flop. Using vesicles similar to the DPPC model vesicle, we will build several new vesicle systems consisting either of POPS and POPC at two different cholesterol concentrations and use these systems to calculate the free energy for cholesterol flip-flop and desorption to investigate the effect of curvature on charged and uncharged membranes. Specifically, we will investigate each vesicle system at at three different temperatures for cholesterol concentrations, leading to a total of six systems. In order to estimate the rate of direct absorption of cholesterol, we will run a series of small, flat membrane simulations where we place a cholesterol molecule just outside the membrane and let it get absorbed without any external influence. Here we will run a set of 25 small simulations measuring the absorption time per system, with two rate estimations per simulation. To investigate the temperature dependence of the absorption rates we will simulate one POPS-cholesterol system and one POPC-cholesterol system at four different temperatures, yielding a total of 8 simulation sets. To calculate the needed free energy profiles we will use the umbrella sampling technique which is particularly useful when proper sampling can not be reached in a simple long equilibrium simulation. To do this, the system is constrained to sample a small region of phase space with a known external potential added to the system potential. The set of constraints used to sample the systems are independent of each other, which means that Umbrella Sampling simulations can be run in parallel. The output sampling data is combined using the Weighted Histogram Analysis method, post simulation. **Estimate of number of core-hours: Direct absorption simulations in the flat membrane systems: 25 × 8 = 200 simulations are needed. With each simulation running on one node of 8 cores at 24 h/simulation (estimated from a benchmark of 0.0064 days/ns) to get enough data, will yield a need for 200 × 24 × 8 = 38,400 ≈ 40,000 core-hours. Vesicle system equilibration simulations: We need to equilibrate the vesicle systems before using them in the free energy calculations. A typical equilibration time is 300 - 400 ns, which with a benchmark time of 0.007 ns/day yields a need for 48 h per simulation. Run on 512 cores, we would need 48 h/simulation × 6 simulation × 512 cores = 147456 ≈ 150,000 core-hours. Cholesterol flip-flop and desorption free energy sampling in the vesicles: 6 systems × 5 simulations/system = 30 sampling simulations are needed. With 30,000 data points per sam- pling region, collected at every 1 ps as a minimum data requirement and with a benchmark time of 0.06 days/ns, we need 32 h per simulation, but 35 - 40 h is desirable. A vesicle simulation typically 600 cores to run at this pace, which yields 30 simulations × 40 hours/simulation × 600 cores ≈ 720,000 core-hours. In total, we need 40, 000 + 150, 000 + 720, 000 ≈ 910,000 core-hours were the 40,000 core hours for the absorption estimate simulations are used on independently run small simulations. We will use the molecular dynamics program NAMD, which is developed at University of Illinois at Urbana-Champaign and is freely available for download (http://www.ks.uiuc.edu/Research/namd/) NAMD is highly parallel and scales efficiently with the number of particles in the system. With the release of version 2.10 of NAMD we have seen an increase in efficiency of about 15 % on other machines for the proposed vesicle systems. Thus we believe that the obtained core hours could be used more efficiently with the new version of NAMD also on the Blues and Fusion clusters. Benchmarks obtained at the UIC Extreme cluster on a 1M coarse-grained particle system: Cores | 80 | 160 | 240 | NAMD 2.9 | 76.9 ns/day | 108 ns/day | 120.7 ns/day | NAMD 2.10 | 83.3 ns/day | 129.8 ns/day | 138.8 ns/day | Current: undetermined amount Justification: The vesicle systems have an obvious advantage of using the clusters at Argonne, since they need hundreds of cores to reach peak efficiency which is something that is far from feasible even on a high end private desktop computer. Current benchmarks used to estimate the core-hours needed for the vesicle system equilibration simulations are from old scaling tests. We have not yet been able to try the new version of NAMD on the blues cluster, but benchmarks obtained at the UIC Extreme cluster on a 1M coarse-grained particle system shows a good scaling up to at least 240 cores and a speedup of 10-20 % compared to NAMD 2.9: Cores | 80 | 160 | 240 | NAMD 2.9 | 76.9 ns/day | 108 ns/day | 120.7 ns/day | NAMD 2.10 | 83.3 ns/day | 129.8 ns/day | 138.8 ns/day | The obvious bottleneck in our project is the umbrella sampling of the vesicle systems. The one order of magnitude slowdown that can be seen when comparing the benchmark numbers between the equilibrium simulations with the umbrella sampling simulations is due to the fact that the force calculations needed for the external constraints applied to the systems during sampling cannot be run in parallel. However, the code used to apply the constraints is user made and we plan on revising e.g. the update frequency of the lists needed to perform the force calculations and the output frequency of our sampled variables. Requested: 30000 A specific reason has been given: Our work has required the usage of all our current allocated simulation time on half the time estimated. We are very close to having a publication and the last of our simulations are critical for finishing our analysis. Given our current usage and the simulations needed to finish the analysis, we ask for 30 000 more core hours to be added to our current allocation. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Blaas 18 Dec '15

18 Dec '15

Granted 100k - JB On Fri, Dec 18, 2015 at 9:47 AM, Bair, Raymond A. <rbair(a)anl.gov> wrote: > Seems OK to me. > > Ray > > > > > > > On 12/17/15, 7:11 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf > of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on > behalf of accounts(a)lcrc.anl.gov> wrote: > >>Hello, >> >>A change in allocation has been requested: >> >> Requester: jlow (John J. Low) >> Project: lcrc-app-engr >> Title: LCRC Application Engineer support work >> Description: This project will be used by the LCRC application >>engineers to do support work for users of the LCRC cluster. This support >>might be in the form of installation and debugging of both LCRC core and >>user applications, benchmarking and performance analysis, user startup >>help, etc. >> >> Current: undetermined amount >>Justification: >> >> Requested: 100000 >> >>A specific reason has been given: >>Someone used up our allocation. I need more time. >> >>This needs to be approved and the final allocation amount decided upon. >> >>Thank You, >>The LCRC Accounts System >>_______________________________________________ >>allocations-admins mailing list >>allocations-admins(a)lcrc.anl.gov >>https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins > > _______________________________________________ > allocations-admins mailing list > allocations-admins(a)lcrc.anl.gov > https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 18 Dec '15

18 Dec '15

Seems OK to me. Ray On 12/17/15, 7:11 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: >Hello, > >A change in allocation has been requested: > > Requester: jlow (John J. Low) > Project: lcrc-app-engr > Title: LCRC Application Engineer support work > Description: This project will be used by the LCRC application >engineers to do support work for users of the LCRC cluster. This support >might be in the form of installation and debugging of both LCRC core and >user applications, benchmarking and performance analysis, user startup >help, etc. > > Current: undetermined amount >Justification: > > Requested: 100000 > >A specific reason has been given: >Someone used up our allocation. I need more time. > >This needs to be approved and the final allocation amount decided upon. > >Thank You, >The LCRC Accounts System >_______________________________________________ >allocations-admins mailing list >allocations-admins(a)lcrc.anl.gov >https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 17 Dec '15

17 Dec '15

Hello, A change in allocation has been requested: Requester: jlow (John J. Low) Project: lcrc-app-engr Title: LCRC Application Engineer support work Description: This project will be used by the LCRC application engineers to do support work for users of the LCRC cluster. This support might be in the form of installation and debugging of both LCRC core and user applications, benchmarking and performance analysis, user startup help, etc. Current: undetermined amount Justification: Requested: 100000 A specific reason has been given: Someone used up our allocation. I need more time. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Blaas 14 Dec '15

14 Dec '15

Ray, The CSAC pages is fixed now and should be updated daily. 150k was granted to the Lattice-QCD project as well. - JB On Thu, Dec 10, 2015 at 6:11 PM, Bair, Raymond A. <rbair(a)anl.gov> wrote: > OK for 150K. > > BTW, the CASC web report shows a date of 12/4. > > > > > > On 12/10/15, 12:12 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf > of John Blaas" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of > wilshire(a)mcs.anl.gov> wrote: > >>Any arguments for not granting this allocation? >> >>- JB >> >>On Fri, Dec 4, 2015 at 9:44 AM, Bair, Raymond A. <rbair(a)anl.gov> wrote: >>> The CSAC report shows a balance of 186K. Is it out of date? >>> >>> >>>https://www.lcrc.anl.gov/jazz/CSAC/project-reports-2016/Summary/summary-d >>>et >>> ails-name-ascending.php?sortOrder=ascending&reportName=name >>> >>> Ray >>> >>> >>> >>> >>> On 12/3/15, 3:48 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf >>>of >>> accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on >>>behalf >>> of accounts(a)lcrc.anl.gov> wrote: >>> >>>>Hello, >>>> >>>>A change in allocation has been requested: >>>> >>>> Requester: sinclair (Donald Sinclair) >>>> Project: Lattice-QCD >>>> Title: Lattice simulations of Conformal and Walking >>>>Technicolor. >>>> Description: We perform simulations to evaluate the functional >>>>integrals of QCD-like >>>>theories formulated on a discrete space-time lattice, to enable >>>>determination >>>>of the non-perturbative aspects these theories. These include the >>>>properties >>>>of these theories at non-zero temperature, including the scales of >>>>confinement >>>>and chiral symmetry breaking, and such zero temperature properties as >>>>spectra >>>>(including the Higgs mass), decay constants and the running of the gauge >>>>coupling constant. We are particularly interested in those theories >>>>where >>>>the >>>>coupling constant evolves very slowly, since these are candidate >>>>'Walking- >>>>Technicolor' theories. Related to these are theories with an infrared >>>>fixed >>>>point (conformal field theories). Our first goal is to differentiate >>>>between >>>>these two different types of behaviour for candidate theories. >>>> >>>>We are performing simulations of QCD-like theories which are models for >>>>Walking or Conformal Technicolor. We have been studying theories which >>>>are >>>>essentially QCD but with colour-sextet rather than colour-triplet >>>>quarks. >>>>We >>>>hope to measure the running of the QCD coupling constant. For 2 or 3 >>>>flavours, >>>>2-loop perturbation theory predicts an infrared fixed point. >>>> >>>>For 2 flavours, it is possible that a chiral condensate forms before >>>>this >>>>fixed point is reached. If so, the fixed point is avoided, the theory is >>>>confining, and chiral symmetry breaks spontaneously. However, there is a >>>>region where the coupling constant evolves very slowly. These are the >>>>properties required for a walking technicolor theory. Simulations we >>>>have >>>>performed so far at finite temperature suggest that this theory might be >>>>conformal. So far, however, the results are inconclusive. >>>> >>>>For 3 flavours we know that the theory should be conformal. Simulations >>>>at >>>>N_t=6 (12^3 X 6 lattice) and N_t=8 (12^3 X 8 lattice), some of which >>>>used >>>>Fusion, did not yet show evidence of conformality. We are now simulating >>>>at >>>>N_t=12 (24^3 X 12 lattice). We started these simulations on Blues, and >>>>intend >>>>to continue them in FY2016 on Blues to enable continuity during the >>>>period >>>>when Edison at NERSC is unavailable due to a site move. >>>> >>>>Our sextet quark codes are based on our earlier triplet quark codes and >>>>use >>>>the RHMC simulation method. The Rational Hybrid Monte Carlo (RHMC) is a >>>>stochastic molecular dynamics algorithm. The functional integral of QCD >>>>is >>>>written as a partition function of a classical field theory evolving in >>>>a >>>>fictitious time. The determinant of the Dirac operator raised to a >>>>fractional >>>>power is calculated by introducing bosonic fields (pseudofermions), and >>>>sandwiching this Dirac operator raised to minus said fractional power >>>>between >>>>them. This fractional power of the Dirac operator is approximated to >>>>machine >>>>accuracy by a rational approximation. After defining this theory on a >>>>discrete >>>>space-time lattice, the inversions required by the partial-fraction >>>>expansion >>>>of the rational approximation are performed using Krylov space methods, >>>>in >>>>particular a multi-shift extension of the conjugate gradient algorithm. >>>>A >>>>global Metropolis Monte-Carlo accept/reject step applied at the end of >>>>each >>>>trajectory removes discretization errors introduced by the numerical >>>>integration of these stochastic equations of motion. We parallelize the >>>>code >>>>by assigning a fixed number of adjacent lattice sites to each MPI task. >>>>Network bandwidth ultimately limits how small a chunk of the lattice can >>>>be >>>>assigned to each task. >>>> >>>>Our 24^3 X 12 runs will be performed on 288 cores (18 nodes) of Blues. >>>> >>>>A short benchmark run of the 24^3 X 12 codes yielded 667 Gflops or 2.3 >>>>Gflops/core. Since we have seen performances of > 3 Gflops/core on >>>>machines >>>>with similar processors, and > 4 Gflops/node on Edison, we suspect that >>>>at >>>>least 3 Gflops/core should be achievable on this machine. Since we have >>>>nearly >>>>exhausted our allocation on Blues, we cannot perform more extensive >>>>tests. >>>>Scaling tests on Fusion and on Edison at NERSC for numbers of cores from >>>>24 up >>>>to 288 cores indicate that the per core performance increases up to 288 >>>>cores, >>>>which we believe to be related to cache usage. >>>> >>>>Our new project simulating QCD at finite mu using Complex Langevin with >>>>Gauge >>>>Cooling is currently running as serial code on several platforms, >>>>including >>>>Blues, where the little allocation we have remaining does not allow us >>>>to >>>>run >>>>anything else. The Langevin equation method is essentially the limiting >>>>case of >>>>the above algorithm in its older form (without the rational >>>>approximation), >>>>when each trajectory involves a single update of the fields. Its >>>>extension to >>>>the complex manifold merely involves extending the gauge-field manifold >>>>from >>>>SU(3) to SL(3,C). The main reason that it is serial code is that this >>>>project >>>>is still in its exploratory phase, meaning that modifications of the >>>>algorithm >>>>are still possible. Converting to parallel MPI code will not be >>>>difficult, >>>>since the compute-intensive kernel is almost identical to that of our >>>>older >>>>QCD codes, as are many other parts of the code. We do, however, expect >>>>some >>>>performance degradation, since we have been running this code at 64-bit >>>>precision, rather than the 32-bit precision of our QCD codes. When we >>>>have >>>>converted to parallel code, we will cease our serial runs on Blues. >>>>While >>>>we >>>>perform zero-temperature simulations at NERSC, TACC and SDSC, we plan to >>>>start >>>>finite temperature simulations on Blues, probably starting on 8^3*4 >>>>lattices >>>>using 16 or 32 cores, and then moving to 12^3*4 and 12^3*6 lattices on >>>>48 >>>>and >>>>72 cores respectively. Note that these runs with low-level parallelism >>>>are >>>>discouraged on the flagship machines at NERSC. >>>> >>>>Our requested allocation is based on running 2 288-core jobs half of the >>>>time >>>>for the first 4-months and jobs using a total of 144 cores half the time >>>>for >>>>the last 8 months. Expected percentage of allocation for serial jobs -- >>>>5%. >>>> Current: undetermined amount >>>>Justification: We have not yet performed a detailed scaling analysis on >>>>Blues, however, we >>>>have such analyses performed on Fusion as well as Edison at NERSC. For >>>>Fusion, >>>>for our QCD with sextet quarks code on a 24^3 X 12 lattice running on >>>>Fusion >>>>we observed the following performances: >>>> 24 cores = 45 Gflops = 1.9 Gflops/core >>>> 48 cores = 95 Gflops = 2.0 Gflops/core >>>> 72 cores = 148 Gflops = 2.1 Gflops/core >>>> 96 cores = 220 Gflops = 2.3 Gflops/core >>>>144 cores = 369 Gflops = 2.6 Gflops/core >>>>288 cores = 784 Gflops = 2.7 Gflops/core >>>>For the same code and lattice size running on Edison at NERSC we >>>>observed >>>>the >>>>following performances. >>>> 24 cores = 62 Gflops = 2.6 Gflops/core >>>> 48 cores = 129 Gflops = 2.7 Gflops/core >>>> 72 cores = 214 Gflops = 3.0 Gflops/core >>>> 96 cores = 330 Gflops = 3.4 Gflops/core >>>>144 cores = 561 Gflops = 3.9 Gflops/core >>>>288 cores = 1219 Gflops = 4.2 Gflops/core >>>>We use a custom assignment of tasks to nodes in order to minimize >>>>communications. Earlier recoding reduced the number of global reductions >>>>in >>>>the routines, which use most of the CPU time, by a factor of 2. >>>> >>>>The new Complex Langevin codes are still serial, but they will be >>>>parallelized >>>>in the same manner as the RHMC codes. Algorithm changes are under >>>>consideration. >>>>There is also the possibility that we will prepare an OpenMP version in >>>>preparation for running on Cori II at NERSC. >>>> >>>> Requested: 150000 >>>> >>>>A specific reason has been given: >>>>Needed to finish current runs. Other resources are not available. >>>>Allocation on Stampede at TACC is exhausted. Allocation on Comet at SDSC >>>>is almost exhausted. NERSC is severely impacted by site move -- Edison >>>>is >>>>offline for 6 weeks, Hopper will be decommissioned 15th December, and >>>>Cori I is inadequate to take up the slack. >>>> >>>>This needs to be approved and the final allocation amount decided upon. >>>> >>>>Thank You, >>>>The LCRC Accounts System >>>>_______________________________________________ >>>>allocations-admins mailing list >>>>allocations-admins(a)lcrc.anl.gov >>>>https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins >>> >>> _______________________________________________ >>> allocations-admins mailing list >>> allocations-admins(a)lcrc.anl.gov >>> https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins >>_______________________________________________ >>allocations-admins mailing list >>allocations-admins(a)lcrc.anl.gov >>https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins > > _______________________________________________ > allocations-admins mailing list > allocations-admins(a)lcrc.anl.gov > https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 10 Dec '15

10 Dec '15

OK for 150K. BTW, the CASC web report shows a date of 12/4. On 12/10/15, 12:12 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of John Blaas" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of wilshire(a)mcs.anl.gov> wrote: >Any arguments for not granting this allocation? > >- JB > >On Fri, Dec 4, 2015 at 9:44 AM, Bair, Raymond A. <rbair(a)anl.gov> wrote: >> The CSAC report shows a balance of 186K. Is it out of date? >> >> >>https://www.lcrc.anl.gov/jazz/CSAC/project-reports-2016/Summary/summary-d >>et >> ails-name-ascending.php?sortOrder=ascending&reportName=name >> >> Ray >> >> >> >> >> On 12/3/15, 3:48 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf >>of >> accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on >>behalf >> of accounts(a)lcrc.anl.gov> wrote: >> >>>Hello, >>> >>>A change in allocation has been requested: >>> >>> Requester: sinclair (Donald Sinclair) >>> Project: Lattice-QCD >>> Title: Lattice simulations of Conformal and Walking >>>Technicolor. >>> Description: We perform simulations to evaluate the functional >>>integrals of QCD-like >>>theories formulated on a discrete space-time lattice, to enable >>>determination >>>of the non-perturbative aspects these theories. These include the >>>properties >>>of these theories at non-zero temperature, including the scales of >>>confinement >>>and chiral symmetry breaking, and such zero temperature properties as >>>spectra >>>(including the Higgs mass), decay constants and the running of the gauge >>>coupling constant. We are particularly interested in those theories >>>where >>>the >>>coupling constant evolves very slowly, since these are candidate >>>'Walking- >>>Technicolor' theories. Related to these are theories with an infrared >>>fixed >>>point (conformal field theories). Our first goal is to differentiate >>>between >>>these two different types of behaviour for candidate theories. >>> >>>We are performing simulations of QCD-like theories which are models for >>>Walking or Conformal Technicolor. We have been studying theories which >>>are >>>essentially QCD but with colour-sextet rather than colour-triplet >>>quarks. >>>We >>>hope to measure the running of the QCD coupling constant. For 2 or 3 >>>flavours, >>>2-loop perturbation theory predicts an infrared fixed point. >>> >>>For 2 flavours, it is possible that a chiral condensate forms before >>>this >>>fixed point is reached. If so, the fixed point is avoided, the theory is >>>confining, and chiral symmetry breaks spontaneously. However, there is a >>>region where the coupling constant evolves very slowly. These are the >>>properties required for a walking technicolor theory. Simulations we >>>have >>>performed so far at finite temperature suggest that this theory might be >>>conformal. So far, however, the results are inconclusive. >>> >>>For 3 flavours we know that the theory should be conformal. Simulations >>>at >>>N_t=6 (12^3 X 6 lattice) and N_t=8 (12^3 X 8 lattice), some of which >>>used >>>Fusion, did not yet show evidence of conformality. We are now simulating >>>at >>>N_t=12 (24^3 X 12 lattice). We started these simulations on Blues, and >>>intend >>>to continue them in FY2016 on Blues to enable continuity during the >>>period >>>when Edison at NERSC is unavailable due to a site move. >>> >>>Our sextet quark codes are based on our earlier triplet quark codes and >>>use >>>the RHMC simulation method. The Rational Hybrid Monte Carlo (RHMC) is a >>>stochastic molecular dynamics algorithm. The functional integral of QCD >>>is >>>written as a partition function of a classical field theory evolving in >>>a >>>fictitious time. The determinant of the Dirac operator raised to a >>>fractional >>>power is calculated by introducing bosonic fields (pseudofermions), and >>>sandwiching this Dirac operator raised to minus said fractional power >>>between >>>them. This fractional power of the Dirac operator is approximated to >>>machine >>>accuracy by a rational approximation. After defining this theory on a >>>discrete >>>space-time lattice, the inversions required by the partial-fraction >>>expansion >>>of the rational approximation are performed using Krylov space methods, >>>in >>>particular a multi-shift extension of the conjugate gradient algorithm. >>>A >>>global Metropolis Monte-Carlo accept/reject step applied at the end of >>>each >>>trajectory removes discretization errors introduced by the numerical >>>integration of these stochastic equations of motion. We parallelize the >>>code >>>by assigning a fixed number of adjacent lattice sites to each MPI task. >>>Network bandwidth ultimately limits how small a chunk of the lattice can >>>be >>>assigned to each task. >>> >>>Our 24^3 X 12 runs will be performed on 288 cores (18 nodes) of Blues. >>> >>>A short benchmark run of the 24^3 X 12 codes yielded 667 Gflops or 2.3 >>>Gflops/core. Since we have seen performances of > 3 Gflops/core on >>>machines >>>with similar processors, and > 4 Gflops/node on Edison, we suspect that >>>at >>>least 3 Gflops/core should be achievable on this machine. Since we have >>>nearly >>>exhausted our allocation on Blues, we cannot perform more extensive >>>tests. >>>Scaling tests on Fusion and on Edison at NERSC for numbers of cores from >>>24 up >>>to 288 cores indicate that the per core performance increases up to 288 >>>cores, >>>which we believe to be related to cache usage. >>> >>>Our new project simulating QCD at finite mu using Complex Langevin with >>>Gauge >>>Cooling is currently running as serial code on several platforms, >>>including >>>Blues, where the little allocation we have remaining does not allow us >>>to >>>run >>>anything else. The Langevin equation method is essentially the limiting >>>case of >>>the above algorithm in its older form (without the rational >>>approximation), >>>when each trajectory involves a single update of the fields. Its >>>extension to >>>the complex manifold merely involves extending the gauge-field manifold >>>from >>>SU(3) to SL(3,C). The main reason that it is serial code is that this >>>project >>>is still in its exploratory phase, meaning that modifications of the >>>algorithm >>>are still possible. Converting to parallel MPI code will not be >>>difficult, >>>since the compute-intensive kernel is almost identical to that of our >>>older >>>QCD codes, as are many other parts of the code. We do, however, expect >>>some >>>performance degradation, since we have been running this code at 64-bit >>>precision, rather than the 32-bit precision of our QCD codes. When we >>>have >>>converted to parallel code, we will cease our serial runs on Blues. >>>While >>>we >>>perform zero-temperature simulations at NERSC, TACC and SDSC, we plan to >>>start >>>finite temperature simulations on Blues, probably starting on 8^3*4 >>>lattices >>>using 16 or 32 cores, and then moving to 12^3*4 and 12^3*6 lattices on >>>48 >>>and >>>72 cores respectively. Note that these runs with low-level parallelism >>>are >>>discouraged on the flagship machines at NERSC. >>> >>>Our requested allocation is based on running 2 288-core jobs half of the >>>time >>>for the first 4-months and jobs using a total of 144 cores half the time >>>for >>>the last 8 months. Expected percentage of allocation for serial jobs -- >>>5%. >>> Current: undetermined amount >>>Justification: We have not yet performed a detailed scaling analysis on >>>Blues, however, we >>>have such analyses performed on Fusion as well as Edison at NERSC. For >>>Fusion, >>>for our QCD with sextet quarks code on a 24^3 X 12 lattice running on >>>Fusion >>>we observed the following performances: >>> 24 cores = 45 Gflops = 1.9 Gflops/core >>> 48 cores = 95 Gflops = 2.0 Gflops/core >>> 72 cores = 148 Gflops = 2.1 Gflops/core >>> 96 cores = 220 Gflops = 2.3 Gflops/core >>>144 cores = 369 Gflops = 2.6 Gflops/core >>>288 cores = 784 Gflops = 2.7 Gflops/core >>>For the same code and lattice size running on Edison at NERSC we >>>observed >>>the >>>following performances. >>> 24 cores = 62 Gflops = 2.6 Gflops/core >>> 48 cores = 129 Gflops = 2.7 Gflops/core >>> 72 cores = 214 Gflops = 3.0 Gflops/core >>> 96 cores = 330 Gflops = 3.4 Gflops/core >>>144 cores = 561 Gflops = 3.9 Gflops/core >>>288 cores = 1219 Gflops = 4.2 Gflops/core >>>We use a custom assignment of tasks to nodes in order to minimize >>>communications. Earlier recoding reduced the number of global reductions >>>in >>>the routines, which use most of the CPU time, by a factor of 2. >>> >>>The new Complex Langevin codes are still serial, but they will be >>>parallelized >>>in the same manner as the RHMC codes. Algorithm changes are under >>>consideration. >>>There is also the possibility that we will prepare an OpenMP version in >>>preparation for running on Cori II at NERSC. >>> >>> Requested: 150000 >>> >>>A specific reason has been given: >>>Needed to finish current runs. Other resources are not available. >>>Allocation on Stampede at TACC is exhausted. Allocation on Comet at SDSC >>>is almost exhausted. NERSC is severely impacted by site move -- Edison >>>is >>>offline for 6 weeks, Hopper will be decommissioned 15th December, and >>>Cori I is inadequate to take up the slack. >>> >>>This needs to be approved and the final allocation amount decided upon. >>> >>>Thank You, >>>The LCRC Accounts System >>>_______________________________________________ >>>allocations-admins mailing list >>>allocations-admins(a)lcrc.anl.gov >>>https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins >> >> _______________________________________________ >> allocations-admins mailing list >> allocations-admins(a)lcrc.anl.gov >> https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins >_______________________________________________ >allocations-admins mailing list >allocations-admins(a)lcrc.anl.gov >https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Blaas 10 Dec '15

10 Dec '15

Any arguments for not granting this allocation? - JB On Fri, Dec 4, 2015 at 9:44 AM, Bair, Raymond A. <rbair(a)anl.gov> wrote: > The CSAC report shows a balance of 186K. Is it out of date? > > https://www.lcrc.anl.gov/jazz/CSAC/project-reports-2016/Summary/summary-det > ails-name-ascending.php?sortOrder=ascending&reportName=name > > Ray > > > > > On 12/3/15, 3:48 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of > accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf > of accounts(a)lcrc.anl.gov> wrote: > >>Hello, >> >>A change in allocation has been requested: >> >> Requester: sinclair (Donald Sinclair) >> Project: Lattice-QCD >> Title: Lattice simulations of Conformal and Walking >>Technicolor. >> Description: We perform simulations to evaluate the functional >>integrals of QCD-like >>theories formulated on a discrete space-time lattice, to enable >>determination >>of the non-perturbative aspects these theories. These include the >>properties >>of these theories at non-zero temperature, including the scales of >>confinement >>and chiral symmetry breaking, and such zero temperature properties as >>spectra >>(including the Higgs mass), decay constants and the running of the gauge >>coupling constant. We are particularly interested in those theories where >>the >>coupling constant evolves very slowly, since these are candidate 'Walking- >>Technicolor' theories. Related to these are theories with an infrared >>fixed >>point (conformal field theories). Our first goal is to differentiate >>between >>these two different types of behaviour for candidate theories. >> >>We are performing simulations of QCD-like theories which are models for >>Walking or Conformal Technicolor. We have been studying theories which are >>essentially QCD but with colour-sextet rather than colour-triplet quarks. >>We >>hope to measure the running of the QCD coupling constant. For 2 or 3 >>flavours, >>2-loop perturbation theory predicts an infrared fixed point. >> >>For 2 flavours, it is possible that a chiral condensate forms before this >>fixed point is reached. If so, the fixed point is avoided, the theory is >>confining, and chiral symmetry breaks spontaneously. However, there is a >>region where the coupling constant evolves very slowly. These are the >>properties required for a walking technicolor theory. Simulations we have >>performed so far at finite temperature suggest that this theory might be >>conformal. So far, however, the results are inconclusive. >> >>For 3 flavours we know that the theory should be conformal. Simulations at >>N_t=6 (12^3 X 6 lattice) and N_t=8 (12^3 X 8 lattice), some of which used >>Fusion, did not yet show evidence of conformality. We are now simulating >>at >>N_t=12 (24^3 X 12 lattice). We started these simulations on Blues, and >>intend >>to continue them in FY2016 on Blues to enable continuity during the period >>when Edison at NERSC is unavailable due to a site move. >> >>Our sextet quark codes are based on our earlier triplet quark codes and >>use >>the RHMC simulation method. The Rational Hybrid Monte Carlo (RHMC) is a >>stochastic molecular dynamics algorithm. The functional integral of QCD is >>written as a partition function of a classical field theory evolving in a >>fictitious time. The determinant of the Dirac operator raised to a >>fractional >>power is calculated by introducing bosonic fields (pseudofermions), and >>sandwiching this Dirac operator raised to minus said fractional power >>between >>them. This fractional power of the Dirac operator is approximated to >>machine >>accuracy by a rational approximation. After defining this theory on a >>discrete >>space-time lattice, the inversions required by the partial-fraction >>expansion >>of the rational approximation are performed using Krylov space methods, in >>particular a multi-shift extension of the conjugate gradient algorithm. A >>global Metropolis Monte-Carlo accept/reject step applied at the end of >>each >>trajectory removes discretization errors introduced by the numerical >>integration of these stochastic equations of motion. We parallelize the >>code >>by assigning a fixed number of adjacent lattice sites to each MPI task. >>Network bandwidth ultimately limits how small a chunk of the lattice can >>be >>assigned to each task. >> >>Our 24^3 X 12 runs will be performed on 288 cores (18 nodes) of Blues. >> >>A short benchmark run of the 24^3 X 12 codes yielded 667 Gflops or 2.3 >>Gflops/core. Since we have seen performances of > 3 Gflops/core on >>machines >>with similar processors, and > 4 Gflops/node on Edison, we suspect that at >>least 3 Gflops/core should be achievable on this machine. Since we have >>nearly >>exhausted our allocation on Blues, we cannot perform more extensive tests. >>Scaling tests on Fusion and on Edison at NERSC for numbers of cores from >>24 up >>to 288 cores indicate that the per core performance increases up to 288 >>cores, >>which we believe to be related to cache usage. >> >>Our new project simulating QCD at finite mu using Complex Langevin with >>Gauge >>Cooling is currently running as serial code on several platforms, >>including >>Blues, where the little allocation we have remaining does not allow us to >>run >>anything else. The Langevin equation method is essentially the limiting >>case of >>the above algorithm in its older form (without the rational >>approximation), >>when each trajectory involves a single update of the fields. Its >>extension to >>the complex manifold merely involves extending the gauge-field manifold >>from >>SU(3) to SL(3,C). The main reason that it is serial code is that this >>project >>is still in its exploratory phase, meaning that modifications of the >>algorithm >>are still possible. Converting to parallel MPI code will not be difficult, >>since the compute-intensive kernel is almost identical to that of our >>older >>QCD codes, as are many other parts of the code. We do, however, expect >>some >>performance degradation, since we have been running this code at 64-bit >>precision, rather than the 32-bit precision of our QCD codes. When we have >>converted to parallel code, we will cease our serial runs on Blues. While >>we >>perform zero-temperature simulations at NERSC, TACC and SDSC, we plan to >>start >>finite temperature simulations on Blues, probably starting on 8^3*4 >>lattices >>using 16 or 32 cores, and then moving to 12^3*4 and 12^3*6 lattices on 48 >>and >>72 cores respectively. Note that these runs with low-level parallelism are >>discouraged on the flagship machines at NERSC. >> >>Our requested allocation is based on running 2 288-core jobs half of the >>time >>for the first 4-months and jobs using a total of 144 cores half the time >>for >>the last 8 months. Expected percentage of allocation for serial jobs -- >>5%. >> Current: undetermined amount >>Justification: We have not yet performed a detailed scaling analysis on >>Blues, however, we >>have such analyses performed on Fusion as well as Edison at NERSC. For >>Fusion, >>for our QCD with sextet quarks code on a 24^3 X 12 lattice running on >>Fusion >>we observed the following performances: >> 24 cores = 45 Gflops = 1.9 Gflops/core >> 48 cores = 95 Gflops = 2.0 Gflops/core >> 72 cores = 148 Gflops = 2.1 Gflops/core >> 96 cores = 220 Gflops = 2.3 Gflops/core >>144 cores = 369 Gflops = 2.6 Gflops/core >>288 cores = 784 Gflops = 2.7 Gflops/core >>For the same code and lattice size running on Edison at NERSC we observed >>the >>following performances. >> 24 cores = 62 Gflops = 2.6 Gflops/core >> 48 cores = 129 Gflops = 2.7 Gflops/core >> 72 cores = 214 Gflops = 3.0 Gflops/core >> 96 cores = 330 Gflops = 3.4 Gflops/core >>144 cores = 561 Gflops = 3.9 Gflops/core >>288 cores = 1219 Gflops = 4.2 Gflops/core >>We use a custom assignment of tasks to nodes in order to minimize >>communications. Earlier recoding reduced the number of global reductions >>in >>the routines, which use most of the CPU time, by a factor of 2. >> >>The new Complex Langevin codes are still serial, but they will be >>parallelized >>in the same manner as the RHMC codes. Algorithm changes are under >>consideration. >>There is also the possibility that we will prepare an OpenMP version in >>preparation for running on Cori II at NERSC. >> >> Requested: 150000 >> >>A specific reason has been given: >>Needed to finish current runs. Other resources are not available. >>Allocation on Stampede at TACC is exhausted. Allocation on Comet at SDSC >>is almost exhausted. NERSC is severely impacted by site move -- Edison is >>offline for 6 weeks, Hopper will be decommissioned 15th December, and >>Cori I is inadequate to take up the slack. >> >>This needs to be approved and the final allocation amount decided upon. >> >>Thank You, >>The LCRC Accounts System >>_______________________________________________ >>allocations-admins mailing list >>allocations-admins(a)lcrc.anl.gov >>https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins > > _______________________________________________ > allocations-admins mailing list > allocations-admins(a)lcrc.anl.gov > https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Blaas 04 Dec '15

04 Dec '15

Ray, The page is now up to date. Going to wait till Monday to confirm all is well to make sure the script that gets fired off every morning actually does so after Adam implemented a fix. - JB On Fri, Dec 4, 2015 at 10:08 AM, John Blaas <wilshire(a)mcs.anl.gov> wrote: > Ray, > >> The CSAC report shows a balance of 186K. Is it out of date? > > Given that the timestamp at the bottom of the page says "Created on: > 2015-11-10T16:31" I would assume so. > > We can look into why those aren't getting generated daily like they should be. > > - JB

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Blaas 04 Dec '15

04 Dec '15

Ray, > The CSAC report shows a balance of 186K. Is it out of date? Given that the timestamp at the bottom of the page says "Created on: 2015-11-10T16:31" I would assume so. We can look into why those aren't getting generated daily like they should be. - JB

1 0