July 2013 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 30 Jul '13

30 Jul '13

Hello, A change in allocation has been requested: Requester: paul (Mark Paul) Project: Chaos Title: Spatiotemporal chaos and turbulence in fluids Description: This is a continuing project with this year's allocation request building upon last year's progress. All of our calculations use spectral element methods to explore the spatiotemporal chaos and turbulence of fluid systems. The code nek5000 has previously demonstrated scalability and efficiency for the specific problems that we are interested in. We continue to make significant progress on the numerical simulation of Rayleigh-Benard convection for experimentally realistic conditions. Rayleigh-Benard convection is the buoyancy induced convection that occurs when a fluid layer is heated uniformly from below. We are most interested in cases where the fluid dynamics of the convection layer is chaotic or turbulent. Specifically, the code integrates the three-dimensional time-dependent Boussinesq equations, any associated reaction-diffusion-advection equations for the transport of scalar quantities of interest, and the linearized tangent-space equations for computing Lyapunov diagnostics. The code is geometrically flexible and can handle a number of experimentally appropriate boundary conditions. The Lyapunov diagnostics we can compute include the spectrum of Lyapunov exponents and vectors that can be used to provide fundamental insights into the complex dynamics that include the fractal dimension and the location of fast growing perturbations. In this allocation we propose to continue exploring the following problems. 1) Transport in chaotic flow fields In this project we are interested in exploring the transport dynamics of a scalar species when the underlying fluid dynamics are spatiotemporally chaotic. There are a number of situations in which this is important and we will explore two in particular. First, is the combustion of premixed gases in a chaotic flow field. Second, is the motion of swimming microorganisms in a fluid layer. Both of these situations can be described by an additional reaction-diffusion-advection equation that is solved simultaneously with the Boussinesq equations that describe the fluid motion of Rayleigh-Benard convection. The nek5000 code already has this apability and our proposed work does not involve significant additional code development. Over the past year we have explored a number of validation cases to ensure the accuracy of the code for the parameter space that we are interested in. We have computed the transport of a passive scalar quantity in a large shallow fluid layer that is undergoing spiral defect chaos. This allowed us to connect our work with previous calculations in the literature (Chiam et al, Physical Review E, 036205, 2005). This has also provided a baseline computation of the passive transport of a scalar quantity. This baseline computation allows for an important for comparison with the cases where the scalar quantity is not passive but active which we will continue to explore in this allocation. A specific case of interest is the combustion of a scalar quantity in a chaotic flow field. We have begun these calculations and validated the basic approach. With this allocation we will explore the parameter space to quantify the transport of an active scalar. In particular we are interested i n cases of small Lewis and large Damkohler numbers that are most closely related to experiment. 2) Quantifying spatiotemporal chaos In this effort we will perform large scale numerical simulations of chaotic fluid convection for the precise conditions of experiment. We will explore large cylindrical domains with aspect ratios of 30 and larger for very long times (several horizontal diffusion times). We are particularly interested in using and developing diagnostics that shed physical insight upon fundamental features of the underlying high-dimensional chaos. We will use our current capability to compute the Lyapunov exponents, Lyapunov vectors, and the fractal dimension (all using the Gram-Schmidt approach) for a range of conditions of current experimental interest in the convection community. One aspect we will focus upon is using the temporal and spatial averages of the leading order Lyapunov vector to quantify regions and dynamics of interest. We are interested in searching for a connection between these Lyapunov based diagnostics, which can only be computed and are not accessible to experiment, with quantities and diagnostics that can be measured in experiment. We will also develop and explore new diagnostics such as covariant Lyapunov vectors to quantify the spatial disorder in the patterns. The covariant Lyapunov vectors are much more difficult to calculate than the Gram-Schmidt based Lyapunov vectors but have the benefit of providing all of the directions of perturbation growth in the high dimensional phase space (called Oseledec's splitting). The covariant Lyapunov vectors have not been calculated for an experimentally accessible fluid system and have the potential to provide new and important insights into the regions and directions that contribute most to the chaos. In order to connect with experiment we are also going to explore the use of realistic boundary conditions such as walls of finite thermal conductivity and varying fluid properties. 3) Turbulent thermal convection. We would like to continue our three-dimensional numerical simulations of fully turbulent Rayleigh-Benard convection cells for higher Rayleigh numbers (10E10) and lower Prandtl number (0.02-0.7). Simulations have never been performed at such high Rayleigh numbers, system size and Prandtl number. This is exactly the region where there is disagreement between theory and experiments, and there is evidence of a trend towards disagreement by our simulation results at lower Rayleigh number. By going to higher Rayleigh number, we will be able to resolve the disagreements and develop a fuller understanding of turbulence. Many experiments are currently being performed in the proposed regime (and even higher Rayleigh number), but this system is not well understood theoretically. Thermal plumes are present in turbulent convection as a means of heat transport. In addition, a large-scale circulation (LSC) has been found, which sweeps the thermal plumes along like leaves in a wind. This LSC is not steady and also undergoes chaotic, abrupt reversals. Visualizations are extremely difficult in experiments, so numerical simulations are essential for bridging the gap between experiment and theory. Heat transport measurements can also be determined from these simulations and compared to experiments and theoretical predictions. Thin thermal and viscous boundary layers are present, and we can perform a detailed analysis of these boundary layers, which are key to understanding the nature of turbulence in these cells. Finally Lyapunov exponents can be computed for these systems and a correlation is expected between the time dependence of the LSC and the time dependence of one of the Lyapunov exponents. Turbulent Rayleigh-Benard systems have application to such diverse phenomena as weather forecasting and the reversal of the earth's magnetic field. URL: More information can be found at the following URL's Janet Scheel http://faculty.oxy.edu/jscheel/research.html Mark Paul http://www.me.vt.edu/mpaul Paul Fischer http://www.mcs.anl.gov/~fischer/ Allocation request: 150,000 Transport in chaotic convection 150,000 Covariant Lyapunov vectors and exponents 150,000 Turbulent thermal convection 450,000 Total Request Current: undetermined amount Justification: N/A Requested: 250000 A specific reason has been given: We would like to request an additional 250,000 core hours on fusion to continue our current projects. All 3 of our current investigations are making very good progress and require more hours to continue. This supplemental allocation should cover these investigations until the next allocation cycle upon which we are planning to submit a renewal. A brief description of the investigations are: 1) Turbulent Rayleigh-Benard convection at very low Prandtl number. 2) Transport of an active scalar in chaotic convection. 3) Using Lyapunov based diagnostics to probe the dynamics of chaotic convection to compare directly with experiment. All of these projects involve student research projects that are in progress and are central to publications that are in progress, and to presentations that will be given at upcoming conferences including the American Physical Society's Division of Fluid Dynamics meeting. These supplemental hours are very important for the progress of these and will allow us to continue to push forward to generate new physical insights. If there is any further information we can provide please do not hesitate to ask. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 13 Jul '13

13 Jul '13

Hello, A change in allocation has been requested: Requester: balaji (Pavan Balaji) Project: radix Title: Scalable Parallel System Software Description: This is a Computer Science request for 49,500 cpu-hours on Fusion for development and evaluation of a family of tools and libraries with a direct benefit to all users of high-performance computing resources. The Radix group in Argonnne's Mathematics and Computer Science division conducts research and development on facilities to better enable efficient Parallel Programming, I/O, and Scientific Understanding. The Fusion system represents an attractive milestone for our software development: projects start out on our breadboard cluster and can further explore scalability in the presence of a high speed network on Fusion before making the jump to BlueGene-scale parallelism. Radix experiments often are concerned with latency. The infiniband network on Fusion provides an excellent low-latency environment for message passing and storage management research while also providing ample bandwidth for I/O and message bandwidth testing. For FY2013 our Messaging middleware experiments include the usual MPICH2 development. We will study the continued improvement of various MPI features including scalability and memory usage, performance tuning, high-level libraries on top of MPI, fault tolerance capabilities, one-sided communication, and collective operations. This allocation will also be used for studying MPI extensions including active messages, and compiler support for MPI refactoring. We also plan for developing application-oriented libraries like ADLB, a load-balancing library. We will port the ADLB load-balancing library to Fusion and test its ability to function with multi-core nodes at intermediate scale, taking advantage of the multiple cores to replace MPI communication with local shared memory. The NWChem library will continue to improve the performance of an important scientific application that is widely used for science by Argonne researchers. Specifically, we will develop inspect-executor load-balancing techniques that will reduce network contention and improve the overall performance and scalability of an important family of methods (coupled-cluster methods). Furthermore, we will explore hybrid programming models (that is, adding threading to the existing GA/ARMCI/MPI model), particularly in the TCE module, which has been used extensively by researchers in Argonne MSD and CSE. All the aforementioned developments will be deployed on Fusion in a production fashion and made available to all users. This allocation will enable Jeff Hammond to maintain the NWChem installations on Fusion, as he has done since the machine went live. We work closely with application scientists, and develop application tools under the 'radix' allocation. For one example, the KMI project defines a high-level standardized interface for computational biology applications relying on distributed search and match semantics for biological reads. We will study efficient data management and data movement techniques to allow scaling to large number of processes. Fusion hosts two quality parallel file systems, GPFS and PVFS. The wider HPC community often asks the Radix file system researchers to compare these file systems head-to-head. Fusion provides us the opportunity to do just that. For FY 2013 our file system research extends into data storage approaches suitable for exascale. Our research into object storage protocols, including high performance access and replication, fits well on Fusion, with the large amount of local storage on compute nodes. Our Exascale storage efforts will explore scalable algorithms for use in next generation HPC storage systems. The Fusion allocation will be used to evaluate programming models, fault detection algorithms, synchronization primitives, and fault tolerance strategies for a prototype object storage system. Our NoLoss project explores how to integrate in-system storage in the I/O software stack. Under the NoLoss project, an abstraction layer for in-system storage was developed. The SCR checkpointing library was modified to store checkpoints locally using this newly developed abstraction layer. The fusion allocation will be used to explore the performance characteristics of this new approach. Hosting two parallel file systems allows us to do more than just drag-race (so to speak). The two file systems have distinct characteristics, and act differently in the broader role of the I/O software stack. On Fusion, we can utilize a single software stack (application, parallel-netcdf, ROMIO), and compare the impact of a change in the underlying file system on overall application behavior. We know from other research, for example, that the two-phase optimization in ROMIO needs to adapt to the underlying file system, and that aligning variables to file system boundaries can yield performance improvements for parallel-netcdf. In FY 2013 we are continuing research into new high level I/O libraries. We anticipate Fusion and Surveyor will be our two main test platforms. Fusion compute nodes actually look fairly attractive to active storage research. These nodes have enough disk space to store non-trivial datasets, and powerful enough processors to carry out complex computations without much impact on storage performance. Radix projects also focus on deriving insight both about application behavior as well as scientific insight. 8 core nodes make efforts like in-situ visualization, where for example an application renders a frame of a movie, ever more feasible. Even without fancy graphics accelerators, the processing power on Fusion nodes makes such analysis and visualization feasible, especially if the visualization can take advantage of multiple cores and multiple nodes. Naturally, the radix visualization tools have demonstrated scalability to Fusion sizes and beyond on Blue Gene, so making full utilization of Fusion should not be a concern. As should be evident from the variety of planned experiments, the Radix group plans to make full use of Fusion. With Argonne employees and student collaborators, we expect around 20 members in FY2013. We plan on about 0% of our jobs being single core jobs. It's crazy that you even have to ask! While Fusion may contain a modest number of compute nodes, the machine still represents an attractive platform for testing: very fast, low latency interconnect; 8 cores per node; .25 TB of storage per compute node. The testing and experiments the Radix group can carry out on Fusion ensures the tools the group develops today will remain relevant even as supercomputers grow. Quite a few radix-developed projects are part of the Fusion software stack: CPU hours for this project yield improvements and benefits not just for the radix group, but for all users of Fusion and indeed users of high-end computational resources worldwide. Current: undetermined amount Justification: Requested: 200000 A specific reason has been given: We need to perform several large scale experiments in a number of our radix system software projects, including improvements to MPI, higher-level programming models, and new runtime systems. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 11 Jul '13

11 Jul '13

Hello, A change in allocation has been requested: Requester: jkim (Jong Kim) Project: POPAD Title: Adjoint Development of POP Ocean Model for Climate Modeling Applications Description: In realizing the importance of the meridional overturning circulation (MOC) to the global climate variations, an essential element is to fully understand the MOC sensitivity to various forcing variables and modeling parameters. However, there are significant uncertainties in the model configurations such as mixing parameter values and initial and boundary conditions. More systematic computational tools are desirable for robust and efficient modeling experiments. Unfortunately, the computational costs to run large-scale sensitivity studies with conventional approaches such as ensemble approaches or perturbed numerical difference are prohibitively high. As an alternative to the conventional approaches, automatic differentiation (AD) has been applied to enhance the POP ocean model with the derivative computations of the TLM/ADM methods. In the previous year, major source codes of the AD-based POP TLM/ADM methods were successfully implemented and optimized for shor t runs of a low resolution model. We plan to continue further applications of the new codes for the sensitivity studies of POP modeling variables with regard to global ocean circulations. In FY2013, the experiments for longer run simulations of the POP ocean models will be mainly performed with 1 degree resolution for the MPI-based parallel run test. It requires to run various types of the CCSM coupled and TLM/ADM simulations. Generally, the multi-decadal integrations of a coupled CCSM system for low resolution problems require a single computational time on the 8 cores of the fusion. The POP TLM/ADM runs take about 10 times of the computational cost compared to the forward POP integrations. Mainly written in Fortran 90, each of modeling component of the CCSM modeling system requires a C compiler and the MPI library to run with a link with Perl script. Current: undetermined amount Justification: Requested: 80000 A specific reason has been given: Current account for POPAD project has a negative allocation balance. There are still a couple months left for the fiscal year of the project and more development test of the POP adjoint code is needed to be exercised. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 02 Jul '13

02 Jul '13

Hello, A change in allocation has been requested: Requester: kclau (Kah Chun Lau) Project: Li_air_battery Title: Computational studies of Li-Air battery components Description: In the coming fiscal year (FY2013), we will complete our current on-going work and explore different systems in Li-air batteries based on experimental input from our colleagues at ANL. To complete the current ongoing study (i.e. ionic transport, ether-based solvents with different Li-salts), our experience indicates that 100, 000 hours is required. For our new study at FY2013, the computational methods will involve largely AIMD (CPMD, VASP) and DFT method (Gaussian09, VASP). For Gaussian09, the DFT calculation scales well up to 8 processors and for 4-6 nodes is the optimal number of nodes for geometry optimization for system size below 100 atoms. We will likely require about 100,000 core hours to study the stability and reaction paths of new longer ether-based (i.e. with spacer and dopants) and DMSO solvents against different Li salts in gas phase and solution phase. To extend our previous DFT study on Li2O2 (crystalline bulk, surfaces and clusters) system, mo re extensive solid phases of Li2O2/LiO2 and Li2O/Li2O2/LiO2 hetero-interfaces (with and without impurity/dopants), together with the equation of states, mechanical, electronic and thermodynamic properties will be explored using DFT planewave VASP code. The VASP code is well-parallelized for 64-72 cores for 100-200 atoms system for a finite k-point calculation. For a single Γ-point calculation with a total atom ~ 300 atoms, a parallel efficiency of ~ 60% -75% can be achieved using 256 cores. Thus to explore a moderately large systems (~ 100-200 atoms), 200 000 core-hour is needed to perform the tasks. In addition, the larger system of combined electrolyte/electrode (electrolyte/Li2O2/carbon with and without metal catalyst) (~ 400 atoms) interfaces will be computed using CPMD and VASP code. We will explore the chemical reactions (i.e. rare events) based on AIMD method. For system involves metal-catalyst (e.g. Au) that involves d-orbital electrons, the blue-moon approach based on VASP code will be used. For a system without metal catalyst, the metadynamics approach implemented in CPMD code will be used. In order to have a reliable statistics and long enough simulation time, we will require about 400, 000 core hours to complete these computational intensive simulations. Current: undetermined amount Justification: For the details and evidence how to use the hours effectively, please see the project description and project report (FY2012). Further details of the scaling and the performance of the codes can be found as follows: (1) VASP: http://cms.mpi.univie.ac.at/vasp/vasp/Performance_parallel_code_on_various_… (2) CPMD: http://cpmd.org/documentation (3) Gaussian09: http://www.gaussian.com/g_prod/g09_glance.htm Requested: 300000 A specific reason has been given: I need to finish up some on-going NEB calculations (120 atom, fix 80 atom) of reactions on surfaces with defects that approximately needs 150000 core-hour, and 5 system of large scale Ab Initio MD calculations (~ 300 atoms) for several production runs that needs 100000 core-hour This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 01 Jul '13

01 Jul '13

Hello, A change in allocation has been requested: Requester: jiechen (Jie Chen) Project: ScalaGAUSS Title: Scalable Gaussian Process Data Analysis Description: ScalaGAUSS is a statistical software for analyzing spatialtemporal data arising from various applications (chemistry, power grids, material science) and modeled as a Gaussian process. The goal of ScalaGAUSS is to handle both gridded and scattered data at the scale of 10^9 to 10^12 points and beyond. Following the framework of a reformulation of the maximum likelihood estimation using sample average approximation, the core of the computations is to solve several linear systems with respect to the fully dense covariance matrix (having the number of rows/columns the same the data size). For gridded data, we implemented a multilevel Toeplitz linear solver and tested it with up to 1024 cores last year; see the FY2012 report. This year’s work will be built on that of the last year. In particular, we will be focusing on 1. Improving the scaling of the solver; 2. Incorporating the solver into ScalaGAUSS. The objective is to finish a working version of ScalaGAUSS that can be applied to real applications, such as for analyzing ozone data. On the other hand, for scatted data, a linear solver requires efficient matrix-vector multiplications. For this, we have developed a fast summation program for a specific covariance kernel, based on the treecode methodology. This program will be the essential building block of the linear solver. We will be investigating its performance for solving linear systems. The goal is to add to ScalaGAUSS a component that handles scattered data as efficiently as for gridded data. Current: undetermined amount Justification: Requested: 100000 A specific reason has been given: Quota used up. New allocation needed for continuing experiments. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0