September 2013 - Allocations-admins - mailman3.cels.anl.gov

Re: [allocations-admins] [LCRC Accounts] Project Request: PETSc-Foam
by Shashi Aithal 27 Sep '13

27 Sep '13

Thanks, Ray. OF 2.2-x needs GCC 4.5.0 and above to work. Also since we are hoping to do some performance monitoring it will be good to run it on Blues initially. We could target some production runs for Fusion at a later time. Dinesh, Jed Brown and Barry from MCS and Prof. Piscaglia from Politecnico di Milano will also be on this project. Shashi ----- Original Message ----- From: "Raymond A. Bair" <rbair(a)anl.gov> To: "LCRC Allocations Admins" <allocations-admins(a)lists.lcrc.anl.gov>, allocations-admins(a)lcrc.anl.gov Sent: Friday, September 27, 2013 9:11:02 AM Subject: Re: [allocations-admins] [LCRC Accounts] Project Request: PETSc-Foam Lets grant the 150K for Q1 and Q2. Can this be done on FUsion? Ray ----------------------------------------- Ray Bair Computing, Environment, and Life Sciences Argonne National Laboratory and the University of Chicago TCS Building 240, Room 4122 9700 South Cass Avenue Argonne, IL 60439 email: rbair(at)anl.gov Phone: (630)252-5751 On 9/27/13 7:11 AM, "accounts(a)lcrc.anl.gov" <accounts(a)lcrc.anl.gov> wrote: >Hello, > >A new project on the LCRC cluster has been requested. Please forward >the information on to the LCRC Allocation sub-committee. > > Applicant's name: Shashi Aithal >Applicant's institution: ANL > Applicant's division: CELS > Project Name: PETSc-Foam > Project title: Investigation of robust, scalable solvers for >OpenFoam > Associated funding: PETSc > Other Systems: > Science: OpenFOAM (OF) is a multi-physics open-source >solver code capable of modeling a wide-range of problems of interest to >various divisions at ANL. One of the strengths of OpenFOAM is that new >solvers and utilities can be created by its users. In this project, we >intend to investigate fast, robust, scalable solvers for problems >involving moving boundaries and combustion. The goal is to study various >methodologies for load-balancing and solvers for large-scale simulations. > Project description: OF is parallel, finite-volume CFD code. OF >version 2.1.0 has been installed and tested on Fusion. Preliminary >scaling studies on dsmcFoam and dieselFoam have shown good scalability. >Small DSMC cases (1 million cells) have shown good scalability up to 192 >cores. DieselFoam has shown good scalability up to 512 cores with about >500K cells. >We intend to use OpenFoam 2.2-x as the starting point in our work. >Various algorithms for efficient domain decomposition and solution >methods for combusting mixtures will be investigated for optimal >performance and robustness. >The first quarter will focus on compiling, running and testing small >problems and studying various methodologies. More detailed cases studies >are planned for the second, third and fourth quarters. > Project URL: > Requested allocation: 350000 > Q1: 50000 > Q2: 100000 > Q3: 100000 > Q4: 100000 > Justification: > > >The requester has used undetermined amount hours of their initial startup >project. > >In addition to approving an initial amount, please specify a Category >and Subcategory for this project. For a list of the current selection >of approved categories, please see: > > https://wiki.lcrc.anl.gov/wiki/Processes/Categories > >Once the Allocation committee has approved the project, please go to >the Project Management page to create it: > >https://accounts.lcrc.anl.gov/projects.php > >Thank You, >The LCRC Accounts System >_______________________________________________ >allocations-admins mailing list >allocations-admins(a)lists.lcrc.anl.gov >https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list allocations-admins(a)lists.lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Request: PETSc-Foam
by Bair, Raymond A. 27 Sep '13

27 Sep '13

Lets grant the 150K for Q1 and Q2. Can this be done on FUsion? Ray ----------------------------------------- Ray Bair Computing, Environment, and Life Sciences Argonne National Laboratory and the University of Chicago TCS Building 240, Room 4122 9700 South Cass Avenue Argonne, IL 60439 email: rbair(at)anl.gov Phone: (630)252-5751 On 9/27/13 7:11 AM, "accounts(a)lcrc.anl.gov" <accounts(a)lcrc.anl.gov> wrote: >Hello, > >A new project on the LCRC cluster has been requested. Please forward >the information on to the LCRC Allocation sub-committee. > > Applicant's name: Shashi Aithal >Applicant's institution: ANL > Applicant's division: CELS > Project Name: PETSc-Foam > Project title: Investigation of robust, scalable solvers for >OpenFoam > Associated funding: PETSc > Other Systems: > Science: OpenFOAM (OF) is a multi-physics open-source >solver code capable of modeling a wide-range of problems of interest to >various divisions at ANL. One of the strengths of OpenFOAM is that new >solvers and utilities can be created by its users. In this project, we >intend to investigate fast, robust, scalable solvers for problems >involving moving boundaries and combustion. The goal is to study various >methodologies for load-balancing and solvers for large-scale simulations. > Project description: OF is parallel, finite-volume CFD code. OF >version 2.1.0 has been installed and tested on Fusion. Preliminary >scaling studies on dsmcFoam and dieselFoam have shown good scalability. >Small DSMC cases (1 million cells) have shown good scalability up to 192 >cores. DieselFoam has shown good scalability up to 512 cores with about >500K cells. >We intend to use OpenFoam 2.2-x as the starting point in our work. >Various algorithms for efficient domain decomposition and solution >methods for combusting mixtures will be investigated for optimal >performance and robustness. >The first quarter will focus on compiling, running and testing small >problems and studying various methodologies. More detailed cases studies >are planned for the second, third and fourth quarters. > Project URL: > Requested allocation: 350000 > Q1: 50000 > Q2: 100000 > Q3: 100000 > Q4: 100000 > Justification: > > >The requester has used undetermined amount hours of their initial startup >project. > >In addition to approving an initial amount, please specify a Category >and Subcategory for this project. For a list of the current selection >of approved categories, please see: > > https://wiki.lcrc.anl.gov/wiki/Processes/Categories > >Once the Allocation committee has approved the project, please go to >the Project Management page to create it: > >https://accounts.lcrc.anl.gov/projects.php > >Thank You, >The LCRC Accounts System >_______________________________________________ >allocations-admins mailing list >allocations-admins(a)lists.lcrc.anl.gov >https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Project Request: PETSc-Foam
by accounts＠lcrc.anl.gov 27 Sep '13

27 Sep '13

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Shashi Aithal Applicant's institution: ANL Applicant's division: CELS Project Name: PETSc-Foam Project title: Investigation of robust, scalable solvers for OpenFoam Associated funding: PETSc Other Systems: Science: OpenFOAM (OF) is a multi-physics open-source solver code capable of modeling a wide-range of problems of interest to various divisions at ANL. One of the strengths of OpenFOAM is that new solvers and utilities can be created by its users. In this project, we intend to investigate fast, robust, scalable solvers for problems involving moving boundaries and combustion. The goal is to study various methodologies for load-balancing and solvers for large-scale simulations. Project description: OF is parallel, finite-volume CFD code. OF version 2.1.0 has been installed and tested on Fusion. Preliminary scaling studies on dsmcFoam and dieselFoam have shown good scalability. Small DSMC cases (1 million cells) have shown good scalability up to 192 cores. DieselFoam has shown good scalability up to 512 cores with about 500K cells. We intend to use OpenFoam 2.2-x as the starting point in our work. Various algorithms for efficient domain decomposition and solution methods for combusting mixtures will be investigated for optimal performance and robustness. The first quarter will focus on compiling, running and testing small problems and studying various methodologies. More detailed cases studies are planned for the second, third and fourth quarters. Project URL: Requested allocation: 350000 Q1: 50000 Q2: 100000 Q3: 100000 Q4: 100000 Justification: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from CESM
by accounts＠lcrc.anl.gov 24 Sep '13

24 Sep '13

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Robert Jacob Project Name: CESM Division: MCS Project title: Community Earth System Model development Associated funding: DOE Other Systems: NCAR, ALCF, OLCF Science: Climate models treat the atmosphere and ocean as interacting fluids. They contain numerical methods to solve the basic partial differential equations which are in turn derived from the Navier-Stokes equations for a fluid on a rotating sphere. They also contain detailed parameterizations to compute the internal and external forcing terms that come from such diverse phenomena as the passage of radiation through the atmosphere, the release of latent heat by phase changes of water and the effects of friction and unresolved turbulent scales. The response of sea ice and the land surface to heating, cooling, precipitation and wind must also be modeled to close the system. The goal of most climate model development is to predict how future climate will change in response to increasing greenhouse gases. The best prediction will probably come from that model which best simulates the current climate. The most straightforward way to increase the fidelity of a climate simulation is to increase the resolution of the grid used in each model's numerical method. But this comes with a computational cost. Another way to increase the fidelity is to find more complete parameterization schemes of the forcing terms and add new parameterization schemes for any missing ones. But this also comes with a computation cost. Development of the worlds leading climate models must strike a balance between available computing power, the resolution of the grid and the completeness of the parameterizations. Project description: The Community Earth System Model (CESM) is a fully-coupled, global climate model that provides state-of-the-art computer simulations of the Earth's past, present, and future climate states. CESM is developed jointly by the National Science Foundation and the Department of Energy. CESM's atmosphere component is the Community Atmosphere Model. CAM currently supports four different numerical cores for solving the basic equations: a spectral transform method, a Semi-Lagrangian scheme, a finite-volume and a finite-element scheme. The ocean component is a finite-difference model called the Parallel Ocean Program and was developed at Los Alamos. CESM's sea ice and land surface models are each among the most complex and complete models in existence. CESM also contains models of chemical and biogeochemical processes within the main components. CESM requires Fortran90 and C compilers and the MPI library to run. We will be using time on Blues to help build a performance model of CESM and to test developments in its software such as the Model Coupling Toolkit and Parallel I/O library. Project URL: http://www.cesm.ucar.edu Current FY Hours Used: undetermined amount New FY Requested allocation: 4000 Q1: 1000 Q2: 1000 Q3: 1000 Q4: 1000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from parvis
by accounts＠lcrc.anl.gov 24 Sep '13

24 Sep '13

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Robert Jacob Project Name: parvis Division: MCS Project title: Parallel Analysis Tools for Ultra-Large Climate Data Sets Associated funding: DOE BER Other Systems: NCAR, NERSC, ALCF, OLCF Science: Like many scientific applications, climate models rely on numerical methods to solve their equations. But insights about the simulated climate are revealed only after performing additional calculations on several years to decades of the multivariate, multidimensional time series output by the climate model. The tools currently used to perform the climate data analysis and visualization (DAV) are serial and can no longer handle the data sizes being generated. We will be developing a new parallel version of a widely used climate DAV system called NCL which will speed up the climate science workflow for researchers throughout the community. Project description: NCL (the NCAR Command Language) is an interpreted scripting language that allows Matlab-like operations on arrays, can read many climate and weather data formats and provides special functions for plotting climate and weather quantities over maps of the earth. NCL produces publication-quality output. To parallelize NCL, we will use two existing parallel libraries. MOAB, the Mesh Oriented Database, contains datatypes and methods for representing unstructured grids in parallel. Intrepid is part of the Trilinos system and is a library for computing operators acting on discretized fields. MOAB and Intrepid both use MPI for their parallelism and MOAB, Intrepid and NCL are written in C++. We will be developing new code to interface NCL with MOAB/Intrepid. Along with developing a parallel version of NCL, we are also investigating the use of task parallelism using the Swift language. We are converting current c-shell scripts that analyze climate data into Swift scripts that can perform the steps in parallel. We are also substituting some of the serial commands in the scripts with parallel versions so the resulting script will be both task- and data- parallel. Are goal is for a scalable DAV tool that can perform all I/O and calculation in parallel while still rending publication quality 2D plots. If we distribute the data evenly among processors in parallel, then all time-domain calculation will be embarrassingly parallel while most spatial calculation will need only nearest neighbor exchanges. Our requested time will be use to test and time the scaling of our software as it is developed. Project URL: http://trac.mcs.anl.gov/projects/parvis Current FY Hours Used: undetermined amount New FY Requested allocation: 4000 Q1: 1000 Q2: 1000 Q3: 1000 Q4: 1000 Justification: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request from foam
by accounts＠lcrc.anl.gov 24 Sep '13

24 Sep '13

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Robert Jacob Project Name: foam Division: MCS Project title: Global Climate Modeling with the Fast Ocean Atmosphere Model Associated funding: Department of Energy, Office of Science National Science Foundation Other Systems: external: NERSC, NCAR Science: Climate models treat the atmosphere and ocean as interacting fluids. They contain numerical methods to solve the basic partial differential equations which are in turn derived from the Navier-Stokes equations for a fluid on a rotating sphere. They also contain detailed parameterizations to compute the internal and external forcing terms that come from such diverse phenomena as the passage of radiation through the atmosphere, the release of latent heat by phase changes of water and the effects of friction and unresolved turbulent scales. In general, climate model development is focused on increasing the fidelity of the simulation compared to the observations. On long time scales, the climate is controlled mostly by the ocean because it is a slower, more dense fluid then the atmosphere and has a higher heat capacity. For some problems in climate science, such as understanding the different climates of the geologic past and the internal variability of the climate, it may not be necessary to have the most accurate representation of the atmosphere in the climate model being used. Project description: Our major objective for FY14 is to continue to test the application of the Ensemble Kalman Filter to FOAM for tuning model parameters in a coupled mode. In spite of decades of effort, coupled ocean-atmosphere general circulation models (OAGCM) like FOAM still suffer from significant model biases, notably in the tropics. A significant part of these biases is caused by uncertainties in the model parameters. So far, however, parameter tuning in OAGCMs remains rudimentary. We will use our time to develop a strategy of a systematic parameter optimization in an OAGCM. In FY13, we have successfully developed adaptive coupled ensemble Kalman Filter system (AcEnKF) for a simultaneous estimation of model parameter and state. We have optimized a parameter – solar penetration depth by assimilating monthly mean observation of sea surface temperature (SST) and sea surface salinity (SSS) in the perfect model framework. Next, we will further investigate AcEnKF in FOAM. Based on our previous work, the parameter estimation is sensitive to ensemble size and some updating technics. And it is also sensitive to the response of the model to the parameters. In FY14, We will further investigate the AcEnKF in FOAM this year: • We will investigate multi-parameter estimations in FOAM. Some atmosphere and coupler parameters will be estimated in our perfect model experiments. • We will further investigate the effect of observation quality and ensemble size to the parameter estimation. • We will further include realistic observations for state and parameter estimation in FOAM • Given the initial condition we get from the first stage, we will do some forecast experiments to check the predictability of longer-term variability in FOAM model. • The second stage we will apply real observations to the system to produce a data assimilation and decadal forecast product. • Good parameter estimation requires good data assimilation. We also need to refine our pure data assimilation skills that will provide support for us to estimate parameter using real observation in the future. For this request, we'll assume fifty members will be used for each experiment. We expect about 150 different sets of experiments for parameter estimation. Each simulation should take 30 years such that the total model years are: 1. Parameter estimation experiments: 50x30x150 = 225k hours Last year, we measured a cost of 2.133 core-hours per simulated year. After improving the efficiency, we reduced the time to about 1 core hour per year in parameter estimation experiments. Besides, some high-frequency experiments are needed for parameter estimation experiments. Still using 50 members, we will conduct 15 cases with the length of 30 years. These high-frequency experiments require 10 core hours per model year. 2. High frequency experiments: 50*15*30*10 = 225k hours We need 5 long data assimilation simulations, about 200 years, to test the performances of our system, and 200 10-year prediction cases to test its ability for ensemble initiation. We will also run some shorter experiments to work on the combination of state estimation and parameter estimation. Data assimilation experiments take 1 core hour per year while prediction run takes 0.3 per model year. 3. AcEnKF and prediction: 50*(200*5+30*100+200*10*0.3)=230k hours Total core hours for experiments: 225+225+230=680k core hours Project URL: http://www.mcs.anl.gov/foam Current FY Hours Used: undetermined amount New FY Requested allocation: 680000 Q1: 170000 Q2: 170000 Q3: 170000 Q4: 170000 Justification: The total experiments are independent and can be run simultaneously. FOAM is very scalable between 4 and 64 processors. We choose to run an individual ensemble member on 16 or 32 because the additional parallelism comes from the ensemble members. We could effectively use the entire machine if it was available. Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for foam
by support＠lcrc.anl.gov 24 Sep '13

24 Sep '13

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2013/?project=foam

1 0

[LCRC Accounts] Yearly Allocation Request from lcrc-app-engr
by accounts＠lcrc.anl.gov 23 Sep '13

23 Sep '13

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: John Valdes Project Name: lcrc-app-engr Division: LCRC Project title: LCRC Application Engineer support work Associated funding: Other Systems: Science: Project description: This project will be used by the LCRC application engineers to do support work for users of the LCRC cluster. This support might be in the form of installation and debugging of both LCRC core and user applications, benchmarking and performance analysis, user startup help, etc. Project URL: http://www.lcrc.anl.gov Current FY Hours Used: undetermined amount New FY Requested allocation: 24000 Q1: 6000 Q2: 6000 Q3: 6000 Q4: 6000 Justification: Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for nano-soft-assembly
by support＠lcrc.anl.gov 23 Sep '13

23 Sep '13

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2013/?project=nano-soft-ass…

1 0

New LCRC report submitted for HNR
by support＠lcrc.anl.gov 22 Sep '13

22 Sep '13

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2013/?project=HNR

1 0