September 2010 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Yearly Allocation Request from ALCF-Bench_dev
by accounts＠lcrc.anl.gov 23 Sep '10

23 Sep '10

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Kalyan Kumaran Project Name: ALCF-Bench_dev Division: ALCF Project title: ALCF Benchmark Development Associated funding: Other Systems: BG/P Science: Project description: We are in the process of the developing a benchmark suite including key kernels from applications of interest to ALCF. We were unable to use our time last year as we had spent most of our time last year extracting these kernels. We now have done so for 8 of the 10 applications identified in the BG/Q SOW. We will be running these kernels on Fusion next year. Also SPEC HPG is developing an OpenMP benchmark suite that we will be testing on Fusion. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 50000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 22 Sep '10

22 Sep '10

Hello, A change in allocation has been requested: Requester: fischer (Paul Fischer) Project: Nek5000 Title: Scalable CFD Algorithms Description: Nek5000 is a petascale fluid/thermal simulation code used for a host of applications including reactor thermal hydraulics, combustion, MHD, oceanography, spatiotemporal chaos, and vascular flow modeling. Nek5000 is a grid-based PDE (partial differential equation) solver in three space dimensions and time. The code has long been at the forefront of high-order methods and scalable algorithms. It was one of the first comprehensive software tools developed for distributed memory parallel platforms. It pioneered the development of high-order discretizations that, because of their ability to accurately capture interacations over a broad range of scales, are now deemed an essential ingredient of petascale simulation science and that are being adapted across virtually all disciplines (e.g., fusion, aeronautics, biofluids and geophysics). We have demonstrated > 80% parallel efficiency on BG/P (strong scaling) out to P=131,072 processors, using only 7300 points/processor at the largest scale. Nek5000 was recognized with a Gordon Bell award in 1999 for scalability and algorithmic quality. Some of the science applications we anticipate addressing in the coming year are: boundary-layer MHD, non-hydrostatic modeling of ocean currents, and film cooling for gas turbine blades. This latter problem is likely to require 50-100K core-hours. Most of the simulations, however, will be performed by the Texas A&M group leading the project with their own compute resources. This is our standard model -- get the problem running, and then turn it over to the research group in charge. In the past, this model has led to very successful collaborations in oceanography ( > 12 publications), spatiotemporal chaos ( > 12 publications), and MHD ( 4 publications and two INCITE awards). We note that we have access to a dedicated 128-core linux cluster (under a nuclear engineering project) and to resource on BG/P through INCITE and discretionay allocations. This request is to address mid-range capabilities -- particularly for those applications that would not fit on the 128-core cluster. Current: undetermined amount Justification: Requested: 350000 A specific reason has been given: Nek5000 is the principal software engine for several projects within MCS, including Petascale Algorithms for Transport Simulation, supported by the DOE Applied Mathematics Program; Advanced Thermal Hydraulics Modeling, funded under the Nuclear Engineear Advanced Modeling and Simulation (NEAMS) program, and others. Access to a midsize machine like Fusion is of significant value in that it allows us to run cases that are too small for (say) BG/P allocations and too larger for local platforms. In addition, it gives us an important test/development platform for support of users around the world who have similar platforms. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from forms
by accounts＠lcrc.anl.gov 22 Sep '10

22 Sep '10

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Gustav Meglicki Project Name: forms Division: MCS Project title: Forms, the co-programmable FDTD solver Associated funding: DoE Other Systems: Teragrid, IU Quarry, ANL MCS workstations Science: Program Forms is a Scheme-extensible FDTD electromagnetic solver, based on Chombo and Guile, and developed for nanophotonics and astrophysics. Project description: The code has been installed and tested--and is now available to ANL Nanophotonics users. This request is for its subsequent occasional maintenance and assistance to users. Whenever changes or additions are made, additional tests may be needed to ensure the code still operates correctly. Project URL: http://perth.ovpit.indiana.edu/gustav/Forms.pdf Current FY Hours Used: undetermined amount New FY Requested allocation: 100000 Justification: Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for cfd_enginemodeling
by support＠lcrc.anl.gov 21 Sep '10

21 Sep '10

Full project report at https://accounts.lcrc.anl.gov/reporting/project/index.pl?project=cfd_engine…

1 0

New LCRC report submitted for cfd_enginemodeling
by support＠lcrc.anl.gov 21 Sep '10

21 Sep '10

Full project report at https://accounts.lcrc.anl.gov/reporting/project/index.pl?project=cfd_engine…

1 0

[LCRC Accounts] Project Request: MINOTAUR
by accounts＠lcrc.anl.gov 21 Sep '10

21 Sep '10

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Ashutosh Mahajan Applicant's institution: ANL Applicant's division: MCS Project Name: MINOTAUR Project title: Next-Generation Solvers for Mixed-Integer Nonlinear Programs: Structure, Search, and Implementation Associated funding: DOE, through grant DE-FG02-05ER25694. Other Systems: None Science: One of our objectives in this project is to develop powerful algorithmic frameworks to solve previously intractable mixed-integer nonlinear programs (MINLPs). Another is to deliver implementations of these frameworks in new, publicly available solvers. Towards these ends, we are developing the solver MINOTAUR that can solve such problems found in many diverse applications. The difficulty and consequently, the solution time for these problems grow exponentially in the size of the input. Our objective is to harness the parallel processing machines to solve these problems in reasonable time. Another important part of this development is experimenting with many different algorithms and parameters in order to identify the best operating parameters based on the problem structure. Project description: We have developed over the last year, a solver named MINOTAUR, that can be used to solve medium size MINLP instances on a single processor. We are now in the stage of making it robust for ill posed problems and also adaptive to the type and structure of the problem. This effort requires extensive testing on a large set (over 300) of instances. Each test requires up to 500 core-hours and many such tests are required. While the first stage described above requires embarrassingly-parallel execution of several single-processor jobs, it is critical for the next stage of making MINOTAUR capable of harnessing distributed memory architectures. MINOTAUR solves MINLPs by a branch-and-bound methods, where the problem is divided recursively into smaller and smaller subproblems. Each subproblem is in turn a small MINLP that can be solved using the serial-processing techniques that have already been developed, but need tuning as described previously. The code being developed is written in C, C++ and Fortran. We are using MPI for the parallel implementation. I have already used my initial start-up allocation in some tests, and all the required libraries and code seem to compile and run fine. Project URL: Requested allocation: 50000 Justification: The requester has used 7519 hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for Comp_Nanocatalysis
by support＠lcrc.anl.gov 20 Sep '10

20 Sep '10

Full project report at https://accounts.lcrc.anl.gov/reporting/project/index.pl?project=Comp_Nanoc…

1 0

New LCRC report submitted for Comp_Nanocatalysis
by support＠lcrc.anl.gov 20 Sep '10

20 Sep '10

Full project report at https://accounts.lcrc.anl.gov/reporting/project/index.pl?project=Comp_Nanoc…

1 0

[LCRC Accounts] Yearly Allocation Request from RIA-RND
by accounts＠lcrc.anl.gov 19 Sep '10

19 Sep '10

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Brahim Mustapha Project Name: RIA-RND Division: PHY Project title: Accelerator and Targets Simulations for the Rare Isotope Accelerator (RIA) facility. Associated funding: LDRD and DOE R and D funding for RIA Other Systems: AlCF Science: This project is two-fold: First, the simulation of different accelerator options for the future RIA facility in order to decide on the most performant and cost-effective option. Second, the simulation of targets to be used for the production of rare isotopes by the ISOL method (Isotope Separation On Line) in order to optimize the geometry and the arrangement of the target material for minimal losses of short-lived isotopes by radioactive decay. Project description: Most of the work proposed last year was postponed but we hope to do it this year: We plan to start simulations of the SNS linac now that the code is updated with all the required beam line elements. The advantage of simulating an existing machine is to be able to benchmark the code against experimental data. This should be a good step forward towards the realization of the concept of the model driven accelerator, that is using a realistic computer model for online machine operations. Project URL: http://www.phy.anl.gov/aebl Current FY Hours Used: undetermined amount New FY Requested allocation: 300000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from batteryQC
by accounts＠lcrc.anl.gov 17 Sep '10

17 Sep '10

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Jeff R. Hammond Project Name: batteryQC Division: LCF Project title: Quantum chemical simulations of batteries and related algorithms and software tools Associated funding: Argonne LDRD, Chemistry Exascale Codesign Center Other Systems: Intrepid/Surveyor: Blue Gene/P at Argonne (INF), Chinook: Opteron/IB at PNNL (8M), Jaguar: Cray XT5 at ORNL (?), Kraken: Cray XT5 at NICS/Teragrid (1M), Ranger: Opteron/IB at TACC/Teragrid (1M), Lincoln: Tesla/Xeon/IB at NCSA/Teragrid (100K), Dirac: Fermi/Nehalem/IB at NERSC (?), Hopper: Cray XE6 at NERSC (?), Palu: Cray XE6 at CSCS (?) Science: (1) quantum chemistry for metal-oxide systems related to battery science (2) statistical mechanics methods for non-equilibrium systems (3) communication runtime systems for HPC networks Project description: Accurate quantum chemistry methods, particularly coupled-cluster theory (CC), will be applied to metal oxide systems in order to understand the chemical properties of batteries. The CC method known CCSD(T) is often called the "gold standard" of quantum chemistry and provides the most accurate results available for a polynomial-scaling algorithm when applied to most chemical systems. In contrast, density-functional theory (DFT) cannot be used reliable for battery-like systems since localized electronic defects create problems for most of the approximate exchange-correlation functionals used today. This project seeks to calibrate quantum chemistry methods for battery-like systems and develop software to enable Argonne's computational investigations in battery science. The second component of this project is the development of massively-parallel non-equilibrium umbrella sampling (NEUS) algorithms (see http://code.google.com/p/parallelsampling) Alex Dickson and Aaron Dinner have developed a theory for this and Alex and Jeff have been implementing this algorithm in parallel using the Global Arrays toolkit. Both software development and applications will be performed using this allocation. We have a poster at Supercomputing 2010 poster accepted on the preliminary results of this work and have submitted an Argonne-UChicago Strategic Collaborative Initiative proposal to continue this work. We have been developing a new communication runtime system for Global Arrays and PGAS languages called OSPRI (formerly known as A1 - Argonne 1-sided - internally, see https://trac.mcs.anl.gov/projects/a1/) The implementation for Blue Gene/P over DCMF is essentially complete and we are now working on an implementation for commodity systems using MPI plus network-specific extensions (e.g. read-modify-write using IB Verbs). Because OSPRI is a drop-in replacement for ARMCI, we will be conducting performance tests of NWChem and Global Arrays using the new runtime. Experiments Planned: We continue to study lithium oxide clusters using coupled-cluster methods in collaboration with the Curtiss group at Argonne and to tune NWChem for Fusion for this and other purposes. The site installation of NWChem is maintained by the PI of this project and testing is conducted as part of this project, which benefits a number of researchers at Argonne. The NEUS experiments are designed by the Dinner group and involve: (1) evaluating the correctness of the NEUS algorithms for various systems, (2) optimizing hybrid MPI-OpenMP parallelism and (3) measuring parallel scalability over IB in comparison to BG/P. All OSPRI experiments are communication benchmarks of one-sided communication from the GA/ARMCI test suite as well as strong-scaling tests of NWChem. Programming Model and Scalability Issues: NWChem relies upon the Global Arrays (GA) library to provide a PGAS programming model which includes one-sided put/get/accumulate and dynamic load-balancing. Scalability issues appear only at the onset of data-starvation, but this is not critical, as efficient strong-scaling can be achieved over more than an order of magnitude variation in node-count, and greater strong-scaling could be achieved if low-end data was available (jobs are memory-bound). Software Requirements: NWChem requires MPI (C bindings only), the OpenIB stack, F77 and C compilers, and an optimized BLAS library. Performance Results and Efficiency of the Code: The NWChem CCSD(T) code hit 92% of theoretical peak on ~1000 Itanium CPUs on the old PNNL supercomputer and ~50% of theoretical peak on close to 100K CPUs of Jaguar. Flop-rate on BlueGene/P is 20-50% of peak due to the small number of registers, other memory limitations, and the absence of true RDMA capability on the BGP interconnect. This is a key motivation for using Fusion. Algorithmic developments are expected to improve the performance of non-BLAS3 operations, such as atomic integral generation. Job Characterization: We expect to run 12-24 hour jobs on 64-256 nodes for production runs and 1-2 hour jobs on 1-16 nodes for software development. While the fat nodes will be helpful in some cases, we are not dependent on them since NWChem scales to the full machine and no data is replicated. Other Information: The expected number of project members is approximately five. Jeff Hammond works on all aspects of the project. Alex Dickson works on NEUS. Jim Dinan will use this allocation for developing OSPRI if necessary. Jeff's summer interns use this project as necessary, but their consumption is nominal. Publications: Over the past six months, this project was used to generate results for the following papers: R. S. Assary, P. C. Redfern, J. R. Hammond, J. Greeley and L. A. Curtiss, Chem. Phys. Lett., accepted (2010). "Predicted Thermochemistry for Chemical Conversion of 5-Hydroxymethyl Furfural." R. S. Assary, P. C. Redfern, J. R. Hammond, J. Greeley and L. A. Curtiss, J. Phys. Chem. B, 114, 9002 (2010). "Computational Studies of the Thermochemistry for Conversion of Glucose to Levulinic Acid." The work on NEUS and OSPRI/A1 has been the subject of oral presentations and posters; conference and journal papers will be submitted in the next six months. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 200000 Justification: NWChem completely parallelized and scales to many thousands of CPUs. The communication mechanism, ARMCI, has been extensively optimized for Infiniband and provides one-sided performance very close to the hardware capacity. The particularly methodology to be used for this project, CCSD(T), is implemented two different ways in NWChem, one of which scales at >90% efficiency from 1024 to 6144 CPUs on the Chinook supercomputer, which is very similar to Fusion, except larger. The other implementation hit ~0.5 PF on 96K cores of JaguarPF and is the basis for a Gordon Bell Finalist by another group of NWChem developers. The serial kernel in CCSD(T) is DGEMM, which should hit 90% of machine peak using Intel MKL. Performance analysis on both Surveyor and Chinook indicate that DGEMM is >50% of the wall time for large jobs and the rest can be tuned to achieve ~10% of theoretical peak. Hence, these should run at ~50% of theoretical peak (50%*90%+50%*10%=50%). The motivation to use Fusion instead of other machines I have access to is that most jobs are memory-bound, so BlueGene/P is not ideal, and the operating systems on both Chinook and Jaguar are unreliable at scale. The NEUS code scales from 1 to 8K cores of BG/P with 99% efficiency. The hybrid parallelism using OpenMP is near perfect on BG/P within a node. On Fusion, we expect less efficient thread parallelism but nonetheless, NEUS will run on the entire Fusion machine if necessary. OSPRI aka A1 has been tested extensively on BG/P and is, by design, almost scale-free. Thank You, The LCRC Accounts System

1 0