February 2017 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 16 Feb '17

16 Feb '17

Hello, A change in allocation has been requested: Requester: borland (Michael Borland) Project: ApsRenewalLattice Title: Lattice Development and Beam Dynamics for APS Upgrade Description: APS has developed a method of using multi-objective genetic algorithms (MOGA) for optimization of accelerator lattices. We have also developed a systematic method of evaluating optimized lattices for robustness, using many ensembles that represent likely outcomes of commissioning. In the coming year, we will refine two multi-bend achromat (MBA) lattice designs that promise to push APS x-ray brightness to world-leading levels. Significant progress has already been made developing a design that is consistent with engineering constraints. The design has been presented at external reviews related to the on-going upgrade project. In FY17, there will be an emphasis on faster iteration with engineering changes, as well as refinement of physics models. A down-selection to a nominal design is expected in early FY17. It is likely that two lattices will be carried forward in the physics analysis to provide a point of comparison and backup in case issues arise with the nominal design (which is expected to be considerably more aggressive than the other candidate). The codes being used for MOGA and ensemble evaluation have all been used in past years on fusion. These include Pelegant, a parallel accelerator simulation code used primarily for particle tracking and acceptance determination, as well as a Tcl script used for genetic optimization. The simulations are close to embarrassingly parallel and thus scale very well. In addition to lattice development, we model the effects of the higher harmonic cavity (HHC), which plays a vital role in increasing the beam lifetime and reducing emittance blow-up. The simulations involve tracking beams consisting of 48 or more bunches of 100,000 or more particles each, and computing the collective electromagnetic interaction of these particles with the vacuum chamber and rf cavities. The methods are well advanced and will be employed with new lattices as they are generated. Detailed modeling has at present been performed mostly for the less aggressive of the two candidate lattices, and will need to be extended to the more aggressive lattice. Alternatives, such as a three-frequency system and low-frequency cavities, will continue to be explored. Beam loss analysis began will continue in FY17. This involves detailed and time-intensive simulation of the injection process and the Touschek scattering process. It also involves evaluation of beam loss collimation schemes, which are needed in order to protect accelerator components and reduce radiation outside the shielding wall. Current: undetermined amount Justification: The primary code being used, ELEGANT, has run successfully on fusion, blues, intrepid, mira, cory, franklin, and other systems. Most simulations are close to embarrassingly parallel, since they are of single-particle dynamics only. Simulations with collective effects show greater than 70% efficiency for 256 or more cores, depending on problem details. Requested: 1000000 A specific reason has been given: The APS Upgrade project is targeting completion of the preliminary design report (PDR) draft by May of this year in time for an external review. Completing the PDR is an FY17 lab goal. We need additional allocation this quarter to complete required calculations for the newly-adopted lattice. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Roberts, John E. 10 Feb '17

10 Feb '17

Great! This time has been granted for radix (120K). -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 2/10/17, 11:17 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of Bair, Raymond A." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of rbair(a)anl.gov> wrote: Pavan says that 120K would be OK, so lets proceed with that. Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 2/6/17, 7:50 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: balaji (Pavan Balaji) Project: radix Title: Scalable Parallel System Software Description: This allocation will be used for optimizations to various systems software components: (1) programming models and runtime systems, (2) data I/O and file systems, (3) fault tolerance, (4) operating systems, and (5) data analysis and visualization. Primary goal of the project is to perform research on improving the core systems software on high-end computing platforms. The radix allocation covers a large number of cross-cutting projects involving subsets of researchers in the group. A few core projects are presented here, but some of the techniques used are shared between them and are not easily separable (e.g., MPI improvements or resilience improvements): 1. Fault Tolerance Techniques: - Local memory management * Advanced SDC Detection through Data Monitoring + Adapt the prediction techniques used in the SZ lossy compressor for SDC detection * Checkpoint size reduction to improve checkpoint/restart performance and energy efficiency + Adapt FTI for using lossy compression of checkpoints * Impact of lossy compression on Checkpoint/restart + Investigate fundamentally and experimentally the relation between discretization/truncation error with lossy compression error * End-to-end Detection of SDCs in workflows featuring a lossy compression stage + Compare combination of SDC detectors and lossy compressors with respect to the accuracy of SDC detection. - Distributed memory management * Integration of checkpointing in workflow environments + Adapt DECAF and/or SWIFT with FTI and perform performance eveluation * MPI Fault Tolerance + Improvements to MPICH to deal with faults * Resilient-MPI and FTI Integration + Experimentation and evaluation of FTI coupled with resilient-MPI 2. Lightweight, topology-aware, and thread-safe communication: - Improvements to MPI+threads * MPICH improvements to play well with threads, including improved locking and lock-free data structures. * Communication progress improvements in the presence of threads. - Lightweight communication * Low instruction-count and low-overhead irregular data movement. - Topology-aware communication * Optimizations to MPI virtual topology functionality 3. Data I/O, RMA and Active Messages: - Dynamic Locality-Aware Workflows * Dynamically distribute the tasks among nodes using Asynchronous Dynamic Load Balancer - Development and Evaluation of Composable Data Services * Exploration of lightweight, customized, domain-specific storage services to improve the performance of data-intensive HPC applications - High Performance Parallel I/O * The ROMIO MPI-IO implementation provides the foundation for many components of the HPC I/O stack. Continue tuning, debugging, and research efforts. 4. Data analysis and visualization: Testing and benchmarking of improvements to data movement infrastructure such as DIY and Decaf software to prepare for next-generation machine architectures and to apply those improvements to applications across the lab that use those software platforms, such as cosmology, climate science, and materials science. The following areas will be tested on Fusion and Blues: - Relaxed BSP synchronization (e.g., for iterative algorithms, to overlap communication with computation and support irregular communication/computation patterns) * irregular communication patterns + short-circuit message completion - Load-balancing algorithms (work stealing, adaptive decompositions) for unstructured time-varying mesh partitions (e.g., reactor design) * work stealing algorithms + adaptive (e.g. k-d tree) domain decompositions - Integration with many-core thread models * Relaxed BSP communication with interchangeable thread models + Worker thread pool for executing block callback functions - Application domain decompositions * Add support for AMR (block-based and patch-based combustion and astrophysics codes) + Add support for directly using existing simulation decompositions (e.g., in climate codes) Current: undetermined amount Justification: 2M core hours requested: We have four core projects. All projects have been demonstrated at scale in the previous years and form the core of the system software stack used in production on the Fusion and Blues machines. We expect to run around ~40 studies for each project on Blues and on Fusion, scaling from 1 node to 256 nodes (average runtime of 1 hour) - Blues: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 16 (cores) x 1 (hour) = 1.3M - Fusion: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 8 (cores) x 1 (hour) = 655K A small fraction of this would be used for debugging. Requested: 200000 A specific reason has been given: We had several very productive students this year who made a lot of progress on their publications. We are still working on the papers, but seem to have run out of core hours. The additional core hours will help us complete the required work for our papers. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 10 Feb '17

10 Feb '17

Pavan says that 120K would be OK, so lets proceed with that. Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 2/6/17, 7:50 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: balaji (Pavan Balaji) Project: radix Title: Scalable Parallel System Software Description: This allocation will be used for optimizations to various systems software components: (1) programming models and runtime systems, (2) data I/O and file systems, (3) fault tolerance, (4) operating systems, and (5) data analysis and visualization. Primary goal of the project is to perform research on improving the core systems software on high-end computing platforms. The radix allocation covers a large number of cross-cutting projects involving subsets of researchers in the group. A few core projects are presented here, but some of the techniques used are shared between them and are not easily separable (e.g., MPI improvements or resilience improvements): 1. Fault Tolerance Techniques: - Local memory management * Advanced SDC Detection through Data Monitoring + Adapt the prediction techniques used in the SZ lossy compressor for SDC detection * Checkpoint size reduction to improve checkpoint/restart performance and energy efficiency + Adapt FTI for using lossy compression of checkpoints * Impact of lossy compression on Checkpoint/restart + Investigate fundamentally and experimentally the relation between discretization/truncation error with lossy compression error * End-to-end Detection of SDCs in workflows featuring a lossy compression stage + Compare combination of SDC detectors and lossy compressors with respect to the accuracy of SDC detection. - Distributed memory management * Integration of checkpointing in workflow environments + Adapt DECAF and/or SWIFT with FTI and perform performance eveluation * MPI Fault Tolerance + Improvements to MPICH to deal with faults * Resilient-MPI and FTI Integration + Experimentation and evaluation of FTI coupled with resilient-MPI 2. Lightweight, topology-aware, and thread-safe communication: - Improvements to MPI+threads * MPICH improvements to play well with threads, including improved locking and lock-free data structures. * Communication progress improvements in the presence of threads. - Lightweight communication * Low instruction-count and low-overhead irregular data movement. - Topology-aware communication * Optimizations to MPI virtual topology functionality 3. Data I/O, RMA and Active Messages: - Dynamic Locality-Aware Workflows * Dynamically distribute the tasks among nodes using Asynchronous Dynamic Load Balancer - Development and Evaluation of Composable Data Services * Exploration of lightweight, customized, domain-specific storage services to improve the performance of data-intensive HPC applications - High Performance Parallel I/O * The ROMIO MPI-IO implementation provides the foundation for many components of the HPC I/O stack. Continue tuning, debugging, and research efforts. 4. Data analysis and visualization: Testing and benchmarking of improvements to data movement infrastructure such as DIY and Decaf software to prepare for next-generation machine architectures and to apply those improvements to applications across the lab that use those software platforms, such as cosmology, climate science, and materials science. The following areas will be tested on Fusion and Blues: - Relaxed BSP synchronization (e.g., for iterative algorithms, to overlap communication with computation and support irregular communication/computation patterns) * irregular communication patterns + short-circuit message completion - Load-balancing algorithms (work stealing, adaptive decompositions) for unstructured time-varying mesh partitions (e.g., reactor design) * work stealing algorithms + adaptive (e.g. k-d tree) domain decompositions - Integration with many-core thread models * Relaxed BSP communication with interchangeable thread models + Worker thread pool for executing block callback functions - Application domain decompositions * Add support for AMR (block-based and patch-based combustion and astrophysics codes) + Add support for directly using existing simulation decompositions (e.g., in climate codes) Current: undetermined amount Justification: 2M core hours requested: We have four core projects. All projects have been demonstrated at scale in the previous years and form the core of the system software stack used in production on the Fusion and Blues machines. We expect to run around ~40 studies for each project on Blues and on Fusion, scaling from 1 node to 256 nodes (average runtime of 1 hour) - Blues: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 16 (cores) x 1 (hour) = 1.3M - Fusion: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 8 (cores) x 1 (hour) = 655K A small fraction of this would be used for debugging. Requested: 200000 A specific reason has been given: We had several very productive students this year who made a lot of progress on their publications. We are still working on the papers, but seem to have run out of core hours. The additional core hours will help us complete the required work for our papers. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by Bair, Raymond A. 10 Feb '17

10 Feb '17

OK, only radix is pending. Ray ----------------------------------------- Ray Bair Argonne National Laboratory On 2/6/17, 7:50 AM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A change in allocation has been requested: Requester: balaji (Pavan Balaji) Project: radix Title: Scalable Parallel System Software Description: This allocation will be used for optimizations to various systems software components: (1) programming models and runtime systems, (2) data I/O and file systems, (3) fault tolerance, (4) operating systems, and (5) data analysis and visualization. Primary goal of the project is to perform research on improving the core systems software on high-end computing platforms. The radix allocation covers a large number of cross-cutting projects involving subsets of researchers in the group. A few core projects are presented here, but some of the techniques used are shared between them and are not easily separable (e.g., MPI improvements or resilience improvements): 1. Fault Tolerance Techniques: - Local memory management * Advanced SDC Detection through Data Monitoring + Adapt the prediction techniques used in the SZ lossy compressor for SDC detection * Checkpoint size reduction to improve checkpoint/restart performance and energy efficiency + Adapt FTI for using lossy compression of checkpoints * Impact of lossy compression on Checkpoint/restart + Investigate fundamentally and experimentally the relation between discretization/truncation error with lossy compression error * End-to-end Detection of SDCs in workflows featuring a lossy compression stage + Compare combination of SDC detectors and lossy compressors with respect to the accuracy of SDC detection. - Distributed memory management * Integration of checkpointing in workflow environments + Adapt DECAF and/or SWIFT with FTI and perform performance eveluation * MPI Fault Tolerance + Improvements to MPICH to deal with faults * Resilient-MPI and FTI Integration + Experimentation and evaluation of FTI coupled with resilient-MPI 2. Lightweight, topology-aware, and thread-safe communication: - Improvements to MPI+threads * MPICH improvements to play well with threads, including improved locking and lock-free data structures. * Communication progress improvements in the presence of threads. - Lightweight communication * Low instruction-count and low-overhead irregular data movement. - Topology-aware communication * Optimizations to MPI virtual topology functionality 3. Data I/O, RMA and Active Messages: - Dynamic Locality-Aware Workflows * Dynamically distribute the tasks among nodes using Asynchronous Dynamic Load Balancer - Development and Evaluation of Composable Data Services * Exploration of lightweight, customized, domain-specific storage services to improve the performance of data-intensive HPC applications - High Performance Parallel I/O * The ROMIO MPI-IO implementation provides the foundation for many components of the HPC I/O stack. Continue tuning, debugging, and research efforts. 4. Data analysis and visualization: Testing and benchmarking of improvements to data movement infrastructure such as DIY and Decaf software to prepare for next-generation machine architectures and to apply those improvements to applications across the lab that use those software platforms, such as cosmology, climate science, and materials science. The following areas will be tested on Fusion and Blues: - Relaxed BSP synchronization (e.g., for iterative algorithms, to overlap communication with computation and support irregular communication/computation patterns) * irregular communication patterns + short-circuit message completion - Load-balancing algorithms (work stealing, adaptive decompositions) for unstructured time-varying mesh partitions (e.g., reactor design) * work stealing algorithms + adaptive (e.g. k-d tree) domain decompositions - Integration with many-core thread models * Relaxed BSP communication with interchangeable thread models + Worker thread pool for executing block callback functions - Application domain decompositions * Add support for AMR (block-based and patch-based combustion and astrophysics codes) + Add support for directly using existing simulation decompositions (e.g., in climate codes) Current: undetermined amount Justification: 2M core hours requested: We have four core projects. All projects have been demonstrated at scale in the previous years and form the core of the system software stack used in production on the Fusion and Blues machines. We expect to run around ~40 studies for each project on Blues and on Fusion, scaling from 1 node to 256 nodes (average runtime of 1 hour) - Blues: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 16 (cores) x 1 (hour) = 1.3M - Fusion: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 8 (cores) x 1 (hour) = 655K A small fraction of this would be used for debugging. Requested: 200000 A specific reason has been given: We had several very productive students this year who made a lot of progress on their publications. We are still working on the papers, but seem to have run out of core hours. The additional core hours will help us complete the required work for our papers. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] Time Added for Projects 2/9/17
by Roberts, John E. 09 Feb '17

09 Feb '17

Yes, those were already done. Thanks! -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov On 2/9/17, 3:04 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of Bair, Raymond A." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of rbair(a)anl.gov> wrote: Thanks, John. There was a new project request: KTP for 86K. Additional time was requested by: AWA-beam-dynamics, 15K OK And, dmft_for_oxides wanted to move 100K from Q3 to Q2 Maybe these have already been taken care of. Ray ----------------------------------------- Ray Bair Argonne National Laboratory On 2/9/17, 2:14 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of Roberts, John E." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of jeroberts(a)anl.gov> wrote: I’ve added the requested time as decided for: GAS_Injection Lattice-QCD uo2_md This should leave only radix as pending. Thanks. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] Time Added for Projects 2/9/17
by Bair, Raymond A. 09 Feb '17

09 Feb '17

Thanks, John. There was a new project request: KTP for 86K. Additional time was requested by: AWA-beam-dynamics, 15K OK And, dmft_for_oxides wanted to move 100K from Q3 to Q2 Maybe these have already been taken care of. Ray ----------------------------------------- Ray Bair Argonne National Laboratory On 2/9/17, 2:14 PM, "allocations-admins-bounces(a)lcrc.anl.gov on behalf of Roberts, John E." <allocations-admins-bounces(a)lcrc.anl.gov on behalf of jeroberts(a)anl.gov> wrote: I’ve added the requested time as decided for: GAS_Injection Lattice-QCD uo2_md This should leave only radix as pending. Thanks. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Time Added for Projects 2/9/17
by Roberts, John E. 09 Feb '17

09 Feb '17

I’ve added the requested time as decided for: GAS_Injection Lattice-QCD uo2_md This should leave only radix as pending. Thanks. -- John Roberts Argonne National Laboratory CELS Systems jeroberts(a)anl.gov

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 06 Feb '17

06 Feb '17

Hello, A change in allocation has been requested: Requester: sinclair (Donald Sinclair) Project: Lattice-QCD Title: Lattice simulations of QCD at finite baryon number density Description: A field theory in the Euclidean time regime can be quantized in terms of a partition function which is the functional integral over the c-number fields of the exponential of minus the action. In the Langevin equation, the fields evolve in a fictitious time such that the derivative of each field with respect to this time is minus the derivative of the action with respect to that field -- the drift term -- plus a gaussian distributed random number appropriately normalized. It can be shown that in the large 'time' limit the fields are distributed with weight exp(-S), where S is the action. The Langevin equation can be extended to the case where S is complex, by replacing the real fields by complex fields. Here it can only be demonstrated that the time averages of observables give the correct expectation values, when the region traversed by the fields is compact and S is holomorphic in the fields. When this Complex Langevin Equation (CLE) is applied to Lattice QCD, the gauge fields must be extended from SU(3) to SL(3,C) and the action involves the gauge fields and their inverses, but not their hermitian conjugates. The fermions are integrated out producing a power of the determinant of the Dirac operator. We obtain an effective action for the gauge fields by using the identity, determinant(M)^q=exp{q*ln[determinant(M)]} =exp{q*Trace[ln(M)]}. The trace is replaced by a stochastic estimator. The derivative with respect to the fields introduces an inverse of the Dirac operator M, which is evaluated using the conjugate gradient method. Unfortunately, since the determinant has zeros, the inverse of M and thus the drift term have poles, so that the drift term is not holomorphic but meromorphic in the fields. It is for this reason that, although the domain over which the fields evolve appears compact, convergence of the observables to their correct values is not guaranteed, and must be checked. We integrate the CLE numerically, inverting the Dirac operator using the conjugate gradient method. The fermion fields are placed on the sites of a 4-dimensional hypercubic lattice and the gauge fields on the links, making the action gauge invariant. If we label the sites by the 4-vector (x,y,z,t), we parallelize using MPI giving each task 1 t coordinate and nz adjacent z coordinates. The x and y coordinates are internal to each task. Each task then performs the same number of operations, and the communications are all nearest neighbour and homogeneous. In particular M only connects nearest-neighbour sites and is sparse. M is discretized using the staggered fermion method. Gauge cooling is implemented to minimize the sum over links of Tr{UU^dagger+(UU^dagger)^[-1]-2}, where U are the lattice gauge fields. Because we use 64-bit floating point (our previous simulations used 32-bit floating point except places where precision is critical), scaling is not as good as is in our previous work. In addition, since we can anticipate algorithmic changes, we have not optimized our conjugate gradient routine as aggressively as before. We plan to do this, as soon as we are satisfied that we have an optimum version. We only have absolute (Gflop) performances for 12^4 lattices measured on NERSC computers. The best such performance was 3.0 Gflops/core on Cori. We assume that the best performances on NERSC machines for 16^4 lattices would be similar. On 64 cores the performance of the code on a 16^4 lattice on the 32 core Haswell nodes of Blues was approximately 10% slower than on Cori. On the 16 core Blues nodes, the performance was approximately 25% slower than on Cori. For scaling studies on Blues, see the Large Allocation Efficiency section. We have performed runs on a 12^4 lattice at beta=6/g^2=5.6 (g is the lattice QCD bare coupling) and quark mass m=0.025, over a wide range of mu values. The results of these runs are in qualitative agreement with what is expected. However, there are quantitative deviations from our expectations. Running with the same beta and m values on a 16^4 lattice for a selection of mu values showed that these deviations are not a finite volume effect. On the 16^4 lattice we are able to simulate at weaker coupling (beta=5.7). At mu=0 we find far better agreement with known results than for beta=5.6. We are extending our beta=5.7 simulations to mu =/= 0 on Blues and Bridges at PSC to determine if there is similar improvement away from mu=0, and propose to continue these runs in FY2017. Note that we have less than 500,000 hours on Bridges until January, 2017. The current runs being performed on Blues are on 16^4 lattices, where we are using 64 cores (2 32-core Haswell nodes). Each job takes 50-60 wallclock-hours. We plan to run similar jobs in FY2017. We intend to run 2 such jobs concurrently, or 1 128 core job. When we are convinced that we have the code we wish to run in extensive production, we will apply optimizations that have proved effective with other codes, to the conjugate gradient routine where the jobs spend most of their time. Project website:www.hep.anl.gov/dks Current: undetermined amount Justification: We estimate that our code achieves approximately 3 Gflops per core on the Haswell nodes for the 64 cores we plan to use for production. 64 cores is the 'sweet spot' for 16^4 lattices where the per core performance is maximum. The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 32-core Haswell nodes on Blues. Each run used all 32 cores on each node, and 1 task/core. 32 cores haswell = 51m44.125s 64 cores haswell = 21m41.861s 128 cores haswell = 13m3.596s 256 cores haswell = 9m43.317s The following table shows wallclock times for identical benchmark runs on various numbers of cores on the 16-core nodes on Blues. Each run used all 16 cores on each node, and 1 task/core. 16 cores = 129m35.228s 32 cores = 52m37.305s 64 cores = 25m35.144s 128 cores = 17m55.801s 256 cores = 16m40.058s As indicated above these numbers would be expected to improve when we apply further optimizations to the conjugate gradient code, which have proved effective on other, similar codes. Our request is based on performing 2 64-core or 1 128-core job continuously, assuming that Blues will be available for production 90% of the time. Requested: 200000 A specific reason has been given: Estimated usage for the rest of this quarter. Project is using time more rapidly than anticipated since our NERSC allocation for this year was significantly less than expected. Blues is now our leading platform for this project. We are making excellent progress and it would be unfortunate if we had to suspend production until April. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 06 Feb '17

06 Feb '17

Hello, A change in allocation has been requested: Requester: balaji (Pavan Balaji) Project: radix Title: Scalable Parallel System Software Description: This allocation will be used for optimizations to various systems software components: (1) programming models and runtime systems, (2) data I/O and file systems, (3) fault tolerance, (4) operating systems, and (5) data analysis and visualization. Primary goal of the project is to perform research on improving the core systems software on high-end computing platforms. The radix allocation covers a large number of cross-cutting projects involving subsets of researchers in the group. A few core projects are presented here, but some of the techniques used are shared between them and are not easily separable (e.g., MPI improvements or resilience improvements): 1. Fault Tolerance Techniques: - Local memory management * Advanced SDC Detection through Data Monitoring + Adapt the prediction techniques used in the SZ lossy compressor for SDC detection * Checkpoint size reduction to improve checkpoint/restart performance and energy efficiency + Adapt FTI for using lossy compression of checkpoints * Impact of lossy compression on Checkpoint/restart + Investigate fundamentally and experimentally the relation between discretization/truncation error with lossy compression error * End-to-end Detection of SDCs in workflows featuring a lossy compression stage + Compare combination of SDC detectors and lossy compressors with respect to the accuracy of SDC detection. - Distributed memory management * Integration of checkpointing in workflow environments + Adapt DECAF and/or SWIFT with FTI and perform performance eveluation * MPI Fault Tolerance + Improvements to MPICH to deal with faults * Resilient-MPI and FTI Integration + Experimentation and evaluation of FTI coupled with resilient-MPI 2. Lightweight, topology-aware, and thread-safe communication: - Improvements to MPI+threads * MPICH improvements to play well with threads, including improved locking and lock-free data structures. * Communication progress improvements in the presence of threads. - Lightweight communication * Low instruction-count and low-overhead irregular data movement. - Topology-aware communication * Optimizations to MPI virtual topology functionality 3. Data I/O, RMA and Active Messages: - Dynamic Locality-Aware Workflows * Dynamically distribute the tasks among nodes using Asynchronous Dynamic Load Balancer - Development and Evaluation of Composable Data Services * Exploration of lightweight, customized, domain-specific storage services to improve the performance of data-intensive HPC applications - High Performance Parallel I/O * The ROMIO MPI-IO implementation provides the foundation for many components of the HPC I/O stack. Continue tuning, debugging, and research efforts. 4. Data analysis and visualization: Testing and benchmarking of improvements to data movement infrastructure such as DIY and Decaf software to prepare for next-generation machine architectures and to apply those improvements to applications across the lab that use those software platforms, such as cosmology, climate science, and materials science. The following areas will be tested on Fusion and Blues: - Relaxed BSP synchronization (e.g., for iterative algorithms, to overlap communication with computation and support irregular communication/computation patterns) * irregular communication patterns + short-circuit message completion - Load-balancing algorithms (work stealing, adaptive decompositions) for unstructured time-varying mesh partitions (e.g., reactor design) * work stealing algorithms + adaptive (e.g. k-d tree) domain decompositions - Integration with many-core thread models * Relaxed BSP communication with interchangeable thread models + Worker thread pool for executing block callback functions - Application domain decompositions * Add support for AMR (block-based and patch-based combustion and astrophysics codes) + Add support for directly using existing simulation decompositions (e.g., in climate codes) Current: undetermined amount Justification: 2M core hours requested: We have four core projects. All projects have been demonstrated at scale in the previous years and form the core of the system software stack used in production on the Fusion and Blues machines. We expect to run around ~40 studies for each project on Blues and on Fusion, scaling from 1 node to 256 nodes (average runtime of 1 hour) - Blues: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 16 (cores) x 1 (hour) = 1.3M - Fusion: 4 (projects) x 40 (studies) x 512 (1 to 256 nodes) x 8 (cores) x 1 (hour) = 655K A small fraction of this would be used for debugging. Requested: 200000 A specific reason has been given: We had several very productive students this year who made a lot of progress on their publications. We are still working on the papers, but seem to have run out of core hours. The additional core hours will help us complete the required work for our papers. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 03 Feb '17

03 Feb '17

Hello, A change in allocation has been requested: Requester: jrussell (John Russell) Project: uo2_md Title: Thermal conductivity of uranium oxide with embedded carbon nanotubes Description: Uranium and uranium oxide spent fuels are stored in water to safely maintain temperatures well below the melting point. Power disruptions from a tsunami interupted cooling of fuel rods at the Fukushima Daichi power plant resulting in a meltdown [1, 2]. Increasing thermal conductivity of fuels will result in a faster cooling rate, which could improve safety. Embedded carbon nanotubes (CNTs) or nanofibers might be a strategy to improve thermal conductivity in bulk UO2. CNTs have large surface areas, high mechanical strength, chemical stability, and thermal conductivity which makes them an intriguing candidate for design of a hybrid material. The goal of this project is calculate structural and thermodynamic properties of UO2 and hybrid materials where experimental values may be unavailable. Our initial approach will be to use a partially ionic model (PIM) of the Born-Mayer-Huggins (BMH) potential to model the UO2 material [3, 4]. The PIM features a reduced ionicity with less than full ionic charge on U and O atoms, and it matches the UO2 lattice constant at temperatures over 1000K, which a full ionic model gets wrong. Likewise, electrostatics will be calculated with a linear-scaling Wolf summation. For coupling terms, C-C interactions will be handled by means of a Tersoff potential [5], and the remaining U-O and U-C interactions will be handled with Morse potentials fit to density functional theory (DFT) reference calculations. With respect to thermodynamic proprerties, we will calculate the thermal conductivity by means of the Muller-Plathe [6] method and the Green-Kubo method. In this way, our approach will capture the salient features of the problem, including ionicity of the UO2 and covalency of the CNTs. Our model systems will include free particles, surfaces and nanocrystalline materials. To calculate properties of our model system, we will conduct molecular dynamics simulations using LAMMPS and DFT calculations will be using CP2K and/or VASP. This modeling approach is well suited for large systems due to the favorable scaling of the potentials and software. LAMMPS is massively parallel with nearly linear scaling with system size which can allow for 100k+ atom models for tens of nanoseconds [7]. CP2K is an electronic structure package with good efficiency for serial and parallel systems and excellent accuracy. [8,9] Depending on the size of the model system, it may make more sense to use VASP with a larger number of K-Points. For this reason, we also include VASP, which is a standard electronic structure package with reasonable parallel scaling up to hundreds of cores. Our team has 5 members: John Russell (MD calculations), John Low (DFT calculations), Peter Zapol, Petr Kral, and Larry Curtiss providing scientific leadership and oversight. [1] D. Cyranoski, G. Brumfiel Nature 477, 139-140 (2011). [2] J. Medalia Congressional Research Service, R41728 (2011). [3] T. Arima, S. Yamasaki, Y. Inagaki, K. Idemitsu J. Alloys Compd. 400, 43-50 (2005). [4] T. Arima, S. Yamasaki, K. Idemitsu, Y. Inagaki J. Nucl. Mater. 376, 139-145 (2008). [5] P. Erhart and K. Albe Phys. Rev. B 71 035211 (2005). [6] F. Muller-Plathe J. Chem. Phys. 106 6082-6085 (1997). [7] S. Plimpton J. Comp. Phys. 117, 1-19 (1995). [8] J. VandeVondele, M. Krack, F. Mohamed, M. Parrinello, T. Chassaing and J. Hutter, Comp. Phys. Comm. 167, 103 (2005). Current: undetermined amount Justification: The total request is for 100k hours, with the bulk of the time spent on DFT calculations of UO2 surfaces and interfaces, which are extremely challenging. Requested: 100000 A specific reason has been given: Would it be possible to get 100k more core-hours for the UO2-MD project? John Lowe and I are working on validating our UO2 DFT computations which are very expensive and difficult. In addition, we are collaborating with Marius Stan and Jianwei Sun to understand why DFT+U calculations of UO2 often do not find the ground state and get stuck in metastable states due to the complexity of f-orbitals in uranium. Jianwei is from the Perdew group and has developed a new meta-GGA exchange-correlation functional called SCAN that should be able to perform much better. Meta-GGA functionals are more expensive, but the additional accuracy will be worth the cost. This method will help our UO2-MD project greatly by providing highly accurate reference calculations so that we can fit a good potential for our carbon-UO2 systems. It will also help Marius in his efforts to explain why metastable states occur when using density functional theory to study the electronic structure of UO2. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0