September 2011 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Yearly Allocation Request from Electromagnetics
by accounts＠lcrc.anl.gov 19 Sep '11

19 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Misun Min Project Name: Electromagnetics Division: MCS Project title: Computational Electromagnetics Associated funding: DOE AMR program and LDRD strategic initiative Other Systems: Argonne BG/P (Intrepid), Jaguar (OLCF) Science: A suite of electromagnetics codes has been under development at MCS/ANL that is based on unstructured hexahedral grids for high accuracy and scalable parallel performance to enable tera- and petascale simulation. The production code NekCEM, a high-order electromagnetic solver that we have developed under DOE projects, is currently capable of scalable simulations up to >131K cores on the Argonne BG/P for solving transport phenomena, including beam dynamics for accelerator devices and light transmissions for nanophotonics devices. To this end, we propose to dramatically increase the capabilities of NekCEM, producing a single, comprehensive, high-fidelity electromagnetic software package that runs in parallel beyond petascale for predictive modeling of nanosensors and photovoltaic solar cells, which will help researchers to discover and explore innovative solutions for current energy shortage, global warming, medical diagnoses, and national security. Project description: The breakthrough goal of this project is to develop NekCEM as an extreme-scalable computational tool for understanding the fundamental physics and predicting optimal designs of next-generation electromagnetic devices. Algorithmic development will include extreme-scale programming using MPI/SMP models; efficient time-stepping algorithms; efficient parallel communication kernels, I/O, visualization, performance enhancement; realistic simulations; and validation of computational results with experimental measurements. We consider a high-order SEDG method with a motivation summarized as follows: fewer grid points per wavelength required to achieve a given accuracy; efficiency, with CPU time depending on the degrees of freedom, not the order of approximation; and geometric flexibility with body-conforming meshes. NekCEM is currently capable of scalable simulations up to >131K cores on leadership-class machines such as the Argonne BG/P. The CPU time per time step is about 0.13 seconds on 131K cores for the case of 1.1 billion grid points, with the scalability achieving 75% efficiency on 131K cores for the number of grid points per core n/P=8,530 and 58% efficiency even with an unrealistically small number of n/P=224. Several I/O approaches have been implemented and extensively tested for the write-performance based on a collective I/O and reduced-blocking I/O for single and multiple outputs on up to 65K cores on BG/P in comparison with one POSIX file per processor (1PFPP). The best I/O rate was achieved from the reduced-blocking I/O with approximately 5 seconds to write a single snapshot of a billion-point simulation. Future tests include runs on OLCF Jaguar and upcoming BG/Q. We expect 4-5 project members. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 40000 Q1: 5000 Q2: 5000 Q3: 15000 Q4: 15000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from UQCLM
by accounts＠lcrc.anl.gov 17 Sep '11

17 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Mihai Anitescu Project Name: UQCLM Division: MCS Project title: Uncertainty Quantification for Land Model in Climate Science Associated funding: Climate Science for a Sustainable Energy Future; PI: Mihai Anitescu; site PI: Ian Foster Other Systems: Science: In this project we study uncertainty quantification (UQ) methods applied to Community Land Model (CLM). CLM is the land model for the Community Earth System Model (CESM) and the Community Atmosphere Model (CAM). It consists of several sets of large and nonlinear complex systems. We aim to develop intrusive uncertainty quantification methods and to demonstrate their computational advantage over a differentiated version of CLM with automatic differentiation (OpenAD) which will open the way for gradient-based optimization for calibration in land testbed. Project description: Notes: This is a renewal of a proposal that we submitted after July. We request an additional 35K hours to what we had allocated last year. Rationale: We have a deliverable on Oct 31, so we expect to burn a lot of time in October. We need about 20K for development, and the rest for production. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 85000 Q1: 25000 Q2: 15000 Q3: 15000 Q4: 30000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from stoch_prog
by accounts＠lcrc.anl.gov 16 Sep '11

16 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Cosmin Petra Project Name: stoch_prog Division: MCS Project title: Stochastic Programming and Application for Energy Systems Associated funding: DOE-ASCR: Mathematics of Petascale Data Program. Scalable Algorithms for Gaussian Processes. Mihai Anitescu, PI. Other Systems: We have obtained and used approx 2 million core hours on Intrepid. We also applied for a similar amount for the next year. Science: This project is an optimization framework that exploits the weather forecast information in the operation of energy systems. Our objective is to illustrate how the use of forecast information and high accuracy physics and computations can translate into lower operating costs. We will solve the stochastic integer unit commitment/energy dispatch problem with network transmission constraints and wind power generation. Project description: In the previous years, we obtained an unprecedented level of parallelization in solving stochastic optimization problems by exploiting the half-arrow shaped structure these problems have and using Elemental to reduce the computational bottleneck associated with the first stage dense Schur complement system. We also implemented a mixed approach MPI+shared memory (SMP) based on IBM's WSMP BG/P sparse solver to obtain intra-node shared memory parallelization. Our C++ code, PIPS, was able to solve a stochastic economic dispatch problem with close to 1 billion variables using 131072 cores on Intrepid with a strong scaling efficiency of more than 90%. A paper on these findings has been accepted to SC11. The final version can be found here: http://www.mcs.anl.gov/~anitescu/PUBLICATIONS/2011/miles-2011-pips-SC.pdf Currently, we only have the capability to solve problems with variables of a continuous nature. In electricity energy systems, however, it is also necessary to deal with binary (integer) variables representing switches and the “on/off” status of generators (a.k.a unit commitment) and transmission lines. In addition, in long-term planning studies, it is necessary to use binary variables to model the addition of transmission lines and generators to satisfy future demands. These optimization problems are called stochastic mixed-integer problems. While it is not uncommon to solve a relaxation of the integer problem that is cheaper to compute, the relaxed solutions are sub-optimal, generate extra-costs in planning the thermal units, reduce the benefits of using wind power and increase the carbon footprint. Furthermore, the relaxed solutions are usually non-integer and the common practice of rounding them to the closer integer causes a failure to satisfy demand and, inevitably , a higher price for energy since expensive electricity from reserves has to be pumped into the grid. Integer optimization problems have a combinatorial nature, and are often solved by recursively exploring the finite but exponentially large binary solution space using an approach called branch and bound. Parallel branch-and-bound has been well studied. An open-source framework, CHiPPS (https://projects.coin-or.org/CHiPPS) has been successfully run on a Blue Gene/L system. However, parallel branch-and-bound has not previously been used to solve problems so large that the continuous relaxation must be decomposed and solved in parallel. We plan to implement such a framework, which combines parallel branch-and-bound, based on existing codes such as CHiPPS, and our code PIPS, thereby obtaining two levels of parallelism. The requested hours on Fusion will be used to run small and medium scale computations with the parallel branch and bound framework we plan to implement. Project URL: http://www.mcs.anl.gov/~petra/pips.html Current FY Hours Used: undetermined amount New FY Requested allocation: 250000 Q1: 25000 Q2: 100000 Q3: 100000 Q4: 25000 Justification: Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for uncertainty-climate
by support＠lcrc.anl.gov 16 Sep '11

16 Sep '11

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2011/?project=uncertainty-c…

1 0

[LCRC Accounts] Project Request: InversedCatalysts
by accounts＠lcrc.anl.gov 16 Sep '11

16 Sep '11

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Yu Lei Applicant's institution: ANL Applicant's division: Energy Systems Project Name: InversedCatalysts Project title: Encapsulating Ultrathin Oxide Layers for Tuning the Catalytic Activity of Noble Metal Nanoparticles Associated funding: Institute for Atom-Efficient Chemical Transformations Other Systems: Science: Thin metal oxide films often exhibit dramatically different chemical and physical properties compared to their bulk counterparts. Especially in the case that the oxide film is only a few Angstroms thick and supported by a metal substrate, the underlying metal substrate often further affects the oxide film properties through charge transfer. Bulk magnesium oxide is frequently used as a support material for heterogeneous catalysts owning to its high thermal stability and surface area. In general, as a non-reducible metal oxide, MgO itself exhibits negligible or low catalytic activity in oxidation reactions. However, it has been predicted by density function theory (DFT) that a single monolayer of MgO supported by a Ag(100) single crystal can activate the adsorbed O2 molecule into a superoxo (O2-) species through the electronic effect. Superoxo species are extremely active in oxidizing CO to CO2 at low temperature. In addition to CO activation, a combin ed low-temperature scanning tunneling microscopy (LT-STM) and DFT study showed that an electronic effect from the underlying metal can also strongly enhance the activity of water dissociation on the MgO surface. In heterogeneous catalysis, metal surfaces are generally considered as the active sites where catalytic reactions take place. Oftentimes, a loss of catalytic activity results from the strong metal support interaction (SMSI) in which the metal surface active sites gradually become covered by the metal oxide support. However, it has been found that a thin oxide film grown on a metal can exhibit much higher catalytic activity than the metal substrate under the same reaction conditions. For instance, a 5 Å FeO(111) film grown on Pt(111) is active for CO oxidation by O2 at 450 K, a temperature far below that at which Pt(111) itself is active. It has been suggested that in the presence of millibar pressures of O2, the bilayer Fe-O film transforms into a trilayer O-Fe-O film which is extremely active and catalyzes CO oxidation according to a Mars-van Krevelen-type mechanism. On the other hand, coordinatively unsaturated ferrous sites at the interface between FeO(111) islands and P t(111) were also found to be very active for activating O2 for CO oxidation. Project description: As discussed in the previous section, an atomic scale understanding of the roles of the composition and structure of the over-coating oxide system in the activation of surface molecules is critical for the rational design of successful catalysts. The structure of thin oxide film comprises of one or only a few monolayers will likely differ substantially from that of the corresponding bulk structure. DFT based on accurate gradient corrected functionals provides the possibility to obtain detailed information in this thickness regime. We will use plane wave density functional calculations to investigate the structure and electronic properties of the interfaces synthesized in this proposal. Another important aspect of the proposed modeling work will be the investigation of reaction pathways. DFT calculations will be used to determine binding energies at different sites at the interfaces for comparison with experiment to determine how to increase activity through reduced poisoning. We will also use these calculations to map out the reaction pathways including transition states and intermediates for key catalytic reactions. Dr. Yu Lei of ES will be the primary invesitgator of this project with Dr. Bin Liu of CNM and Dr. Neil M. Schweitzer of CSE as the co-investigators. To complete the above work, we are requesting 400,000 core hours for FY2012. Project URL: Requested allocation: 400000 Q1: 100000 Q2: 100000 Q3: 100000 Q4: 100000 Justification: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for LiBattModeling
by support＠lcrc.anl.gov 16 Sep '11

16 Sep '11

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2011/?project=LiBattModeling

1 0

New LCRC report submitted for Dissolution
by support＠lcrc.anl.gov 16 Sep '11

16 Sep '11

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2011/?project=Dissolution

1 0

[LCRC Accounts] Yearly Allocation Request from POPAD
by accounts＠lcrc.anl.gov 15 Sep '11

15 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Jong Kim Project Name: POPAD Division: MCS Project title: Adjoint Development of POP Ocean Model for Climate Modeling Applications Associated funding: DOE CCPP Program: IMPACTS Project Other Systems: Science: The main purpose of the DOE IMPACTS project is to quantify the risks of abrupt climate change during the 21st century. The basic research effort is focused on assessing the near-term prospects for abrupt climate change on decadal time scales. The investigation is based on the NCAR-DOE Community Climate System Model (CCSM). It involves various types of new developments to examine several potentially significant sources of abrupt climate change. The role of ANL is to enhance the CCSM POP Ocean model with tangent linear and adjoint modeling (TLM/ADM) capabilities to allow a detail sensitivity analysis of the POP ocean model. Project description: In realizing the importance of the meridional overturning circulation (MOC) to the global climate variations, an essential element is to fully understand the MOC sensitivity to various forcing variables and modeling parameters. However, there are significant uncertainties in the model configurations such as mixing parameter values and initial and boundary conditions. More systematic computational tools are desirable for robust and efficient modeling experiments. Unfortunately, the computational costs to run large-scale sensitivity studies with conventional approaches such as ensemble approaches or perturbed numerical difference are prohibitively high. As an alternative to the conventional approaches, automatic differentiation (AD) has been applied to enhance the POP ocean model with the derivative computations of the TLM/ADM methods. In the previous year, major source codes of the AD-based POP TLM/ADM methods were successfully implemented and optimized f or short runs of a low resolution model. We plan to continue further applications of the new codes for the sensitivity studies of POP modeling variables with regard to global ocean circulations. In FY2012, the experiments for longer run simulations of the POP ocean models will be mainly performed with both of low and high resolution problem sets. It requires to run various types of the CCSM coupled and TLM/ADM simulations. Generally, the multi-decadal integrations of a coupled CCSM system for low resolution problems require a single computational time on the 8 cores of the fusion. The POP TLM/ADM runs take about 10 times of the computational cost compared to the forward POP integrations. Mainly written in Fortran 90, each of modeling component of the CCSM modeling system requires a C compiler and the MPI library to run with a link with Perl script. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 40000 Q1: 10000 Q2: 10000 Q3: 10000 Q4: 10000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from Climate_Emulator
by accounts＠lcrc.anl.gov 15 Sep '11

15 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Robert Jacob Project Name: Climate_Emulator Division: MCS Project title: Building a climate emulator for use in energy policy analysis Associated funding: NSF, UC-Argonne Seed Grant. Other Systems: Science: State-of-the-art general circulation models (GCMs) provide the best estimates of future climate change. While they are valuable tools for assessing impacts of climate change for a given scenario of future greenhouse gas emissions, they cannot be used in iterative economic models since a single multi-century simulation can take weeks to months to run and the iterative models require hundreds of thousands of iterations. For this reason, we aim to develop a statistical emulator that will accurately and efficiently reproduce GCM forecasts of regional climate change for an arbitrary trajectory of CO2 concentrations by using a library of pre-computed GCM output. Library based statistical emulation has not been possible to this date because there is no consistent and comprehensive library of climate model runs corresponding to a range of scenarios of future CO2 concentrations. Computing time on Fusion will enable us to build this unique resource by performing a series of s imulations using the Community Climate System Model (CCSM3). Since library-based statistical emulation is currently not possible, the most common approach to emulating regional climate uses linear pattern scaling, in which changes in regional temperature and precipitation are assumed to be linearly proportional to the change in global mean temperature (GMT). That is, regional climate is effectively assumed to depend solely on GMT, and not on the rate or duration of warming. Our preliminary analysis of CCSM3 output from the Climate Model Intercomparison Project (CMIP3) archive shows that this assumption is not valid for temperature projections over time scales of multiple centuries and breaks down for projections of precipitation even over decadal time scales. The CMIP3 archive, however, is not sufficient for full statistical emulation, since it is both restricted in number of runs and inconsistent in its treatment of regional climate forcing. The carefully designed library we propose here will be used to develop and test a number of alternative approaches for emulating climate model responses, including (i) nearest neighbor interpolation, (ii) a variant of pattern scaling that accounts for the rate of warming, and (iii) principal component analysis (PCA). The intended outcome of this research is a new climate emulator that will aid in the evaluation of energy policies for greenhouse gas mitigation. We anticipate that there will be a wide interest in a user-friendly, computationally tractable emulation tool from the policy, economics, and climate communities. The library itself will also be made public for further use by the community Project description: To assist in analysis of climate-related policies, we have pioneered the technique of library-based climate model emulation, in which a library of pre-computed model runs is used to generate a statistical tool that reproduces the output of complex models in a fraction of the time it takes to run the full model. Because no existing archive of climate model runs is appropriate for this purpose, building a new climate library is a prerequisite for developing and testing a climate model emulator. We used Fusion computing time in FY2011 to develop a prototype climate library from runs of CCSM3 (the open-source Community Climate System Model 3 built by the National Center for Atmospheric Research), and are applying here for computing time to continue that library development. This library represents a substantial advance over the largest extant archive of climate model runs, the CMIP3 archive that holds the runs incorporated in the periodic reports of the Intergovernmental Panel on Climate Change. The CMIP3 archive contains only three scenarios for future CO2 concentration growth, does not study CO2 growth beyond 2100, and considers different regional sulphate aerosol emissions for each scenario, complicating any understanding of regional climate patterns expected in a warming world. The library we are developing is consistent and comprehensive, with a large number of scenarios, multiple realizations of each scenario, and multi-century simulations. It will serve as tool both for basic science and to make possible new emulation methods. The dominant computational requirements in CCSM3 are its atmosphere and ocean components. The atmosphere component is the Community Atmosphere Model, which currently supports three different numerical cores for solving the basic equations: a spectral transform method, a Semi-Lagrangian scheme, and a finite-volume scheme. The ocean component is a finite-difference model called the Parallel Ocean Program and was developed at Los Alamos. CCSM3 requires Fortran90 and C compilers and the MPI and NetCDF libraries to run. In FY2011, computing time on Fusion was used to build a library of CCSM3 runs with T31 (3.5ox3.5o) resolution. These runs have been used to develop and test a climate model emulator that accurately replicates future temperate and precipitation projections within the intrinsic uncertainty of the model. In FY2012 we will expand our climate library to explore the importance of parametric uncertainty on climate model projections (especially in parameters relating to cloud coverage and water content, which have been shown to be key drivers of uncertainty in climate projections). The work bears some similarities to the well-known “climateprediction.net” project which used community-donated computing time over many years to do a similar experiment on a higher-resolution version of a British climate model. We have found however that similar physical insights can be recovered at more modest computing requirements by lowering the resolution to T31. We will perform T31 CCSM3 simulations with three values for each of three cloud parameters (nine combinations). For each combination we will spin-up the climate using a 500 year control run and a 140 historical forcing run (1870-2010). Based on results from our FY2011 simulations, we will require three scenarios of future CO2 concentrations from 2010-2160, with three realizations of each scenario, in order to “train” our statistical model. To test the emulator we will require another two scenarios, but will consider only one realization of each. The total requirements for this experiment are 9 * (500 + 140 + 3*150*3 + 2*150) = 21,000 model years. The T31 resolution version of CCSM3 requires 38 core hours per model year, which gives a total of 21,000*38 or approximately 800,000 core hours. In FY2011 we used approximately 550K core hours from an allocation of 900K. Our ability to use our entire allocation was limited by long queue times on Fusion, and having to run a large number of simulations sequentially. In the FY2012 experiments we will be running more simulations in parallel, as simulations for different parameter combination will be run at the same time, allowing for higher usage rates. We are storing data on the Argonne/UChicago PADS system and so have sufficient disk space to store the output offsite. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 800000 Q1: 200000 Q2: 200000 Q3: 200000 Q4: 200000 Justification: Our allocations so far have allowed us to perform thousands of years of simulations using CCSM3 at T31. This provides us with detailed statistics on the time required to run this model configuration on Fusion. Using 8 nodes (64 processors) takes an average of 38 core-hours for a simulation of one model year. Several simulations can be run at the same time to scale the project out to the full machine if available. CCSM3 was the recipient of a great deal of tuning and evaluation by combined DOE and NCAR computational scientists so we do not expect further improvements in the per node performance. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from foam
by accounts＠lcrc.anl.gov 15 Sep '11

15 Sep '11

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Robert Jacob Project Name: foam Division: MCS Project title: Global Climate Modeling with the Fast Ocean Atmosphere Model Associated funding: Department of Energy, Office of Science National Science Foundation Other Systems: external: NERSC, NCAR Science: Climate models treat the atmosphere and ocean as interacting fluids. They contain numerical methods to solve the basic partial differential equations which are in turn derived from the Navier-Stokes equations for a fluid on a rotating sphere. They also contain detailed parameterizations to compute the internal and external forcing terms that come from such diverse phenomena as the passage of radiation through the atmosphere, the release of latent heat by phase changes of water and the effects of friction and unresolved turbulent scales. In general, climate model development is focused on increasing the fidelity of the simulation compared to the observations. On long time scales, the climate is controlled mostly by the ocean because it is a slower, more dense fluid then the atmosphere and has a higher heat capacity. For some problems in climate science, such as understanding the different climates of the geologic past and the internal variability of the climate, it may not be necessary to have the most accurate representation of the atmosphere in the climate model being used. Project description: Our major objective is to test the application of the Ensemble Kalman Filter to FOAM for tuning model parameters in a coupled mode. In spite of decades of effort, coupled ocean-atmosphere general circulation models (OAGCM) like FOAM still suffer from significant model biases, notably in the tropics. A significant part of these biases is caused by uncertainties in the model parameters. So far, however, parameter tuning in OAGCMs remains rudimentary. We will use our time to develop a strategy of a systematic parameter optimization in an OAGCM. In Fy11, we have successfully implemented coupled EnKF data assimilation system on FOAM on FUSION. Next, we use this system to develop an adaptive coupled ensemble Kalman Filter system (AcEnKF) for a simultaneous estimation of model parameter and state. We will also explore different assimilation techniques to search for an effective method of parameter optimization in OAGCMs. Given the time, we can also study coupled dynamics relevant to parameter optimization in the tropical climate system for both the climatology and climate variability. Based on our previous work, we found that the data assimilation results are very sensitive to some parameters of the ensemble package. We need to do some tests to find the optimal parameters for the ensemble package first. Then we will move on to parameter estimation. For this request, we'll assume fifty members will be used for each experiment. We expect about 150 different sets of parameters and initial conditions. Each integration should take 30 years such that the total model years are: Parameter estimation experiments: 50x30x150 =225,000 years. We will also test a new Ensemble Adjustment Kalman Filter scheme that we developed recently. In this scheme, ensemble members will be randomly grouped into subgroups at each assimilation time. Simple model study shows that this scheme reduces outliers and therefore reduces analysis error. We will be testing this scheme in FOAM. We will test 60 experiments, each of 30 model years. This gives an additional model years: Random grouping EAKF experiments: 50x30x60 =90,000 years We have measured a cost of 2.133 core-hours/simulated year leading to a request of 640000 core-hours. This gives us the total hours for model experiments as Total cpu for experiments: (225000+90,000)yearsx2.133 = 692K core-hours With some minor time for testing, we request a total of 700k core-hours for FY12. Project URL: http://www.mcs.anl.gov/foam Current FY Hours Used: undetermined amount New FY Requested allocation: 700000 Q1: 150000 Q2: 150000 Q3: 150000 Q4: 250000 Justification: We have measured a cost of 2.133 core-hours/simulated year. The 210 total experiments are independent and can be run simultaneously. FOAM is very scalable between 4 and 64 processors. We choose to run on 16 or 32 because the additional parallelism comes from the ensemble members. Thank You, The LCRC Accounts System

1 0