August 2017 - Allocations-admins - mailman3.cels.anl.gov

[LCRC Accounts] Project Request: REMM
by accounts＠lcrc.anl.gov 28 Aug '17

28 Aug '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Jason Clark Applicant's institution: ANL Applicant's division: PHY Project Name: REMM Project title: Reverse engineering nuclear mass models from the rare earth peak of the r-process solar abundance pattern Associated funding: Rebecca/Nicole funding: U.S. Department of Energy Office of Science under Contract No. DE-AC52-07NA27344 for the topical collaboration Fission In R-process Elements (FIRE). Rebecca/Gail funding: U.S. Department of Energy under grant numbers DE-SC0013039 (RS) and DE-FG02-02ER41216 (GCM) and the National Science Foundation under Grant No. PHY-1430152 (JINA Center for the Evolution of the Elements). Matt funding: National Nuclear Security Administration of the U.S. Department of Energy at Los Alamos National Laboratory under Contract No. DE-AC52-06NA25396 (MM). Other Systems: University of Notre Dame Center for Research Computing (two dedicated 24 core machines; unlimited core-hours) Science: The heaviest elements are formed in nature via the process of rapid neutron capture (r-process). Dynamical astrophysical calculations of the abundances produced during r-process nucleosynthesis require knowledge of nuclear data, such as nuclear masses, far from stability where little to no experimental data exists. Therefore these calculations must make use of theoretical models which can produce abundance patterns inconsistent with observational data. We therefore invert the problem by using the observational data for the rare earth peak of the r-process abundance pattern to find the nuclear masses required to fit this region. To perform this fit we use the Metropolis-Hastings algorithm to make predictions for the mass corrections to the Duflo-Zuker mass model. Computational power is necessary for these studies since updates to masses, beta-decay rates, and neutron capture rates must be performed at each time step in order to calculate the correspon ding abundance prediction in a self-consistent manner. This procedure is performed for a variety of astrophysical environments, including neutron star mergers and supernovae-like trajectories, in order to find the trends in the mass surface required to fit the rare earth solar data. Then using the Penning trap mass measurement data for neutron rich nuclei, such as neodymium-160, recently measured at ANL with CARIBU, we examine whether our mass predictions for a given astrophysical environment are consistent with these mass measurements. Thus this work can potentially rule out certain astrophysical sites from being considered the dominate site of production for the heaviest elements. Project description: The computational method employed by the code is the Markov Chain Monte Carlo (MCMC) method which we use to probe the parameter space of nuclear mass models. We use python modules to randomly vary twenty-seven nuclear model parameters at a given time step in order to determine mass corrections to the Duflo-Zuker mass model. Given twenty-seven model parameters, we have a large parameter space to explore. After these adjustments to nuclear masses, we then recalculate all the astrophysical rates (i.e. neutron capture rates and beta decay rates) for the hundreds of nuclei involved in the formation of the rare earth peak. To ensure that the Monte Carlo step is completed in a timely fashion, we parallelize the computation of the astrophysical rates. We perform the parallelization dynamically based off the number of cores available on a computing node, so the code need not be adjusted to run on Argonne machines. We use python wrappers to interface with the Fortran/C++ based code (BeoH) which calculates beta decay rates and beta-delayed neutron emission probabilities. We then use python modules and wrappers to write the nuclear input files and control file for our fortran based nucleosynthesis code (PRISM) (also parallelized). We then make use of python modules to determine the likelihood of inputs via the Metropolis-Hastings algorithm which determines the acceptance/rejection of a step. For a given set of astrophysical conditions, we must perform at least 10 independent runs to have the statistics required to make use of the "parallel chains" MCMC approach which ensures our average values and error bands come from uncorrelated results. With the call to BeoH for the nuclear rates and probabilities, followed by the PRISM nucleosynthesis calculation, each time step takes ~(10-60) seconds depending on the number of cores available on the computing node. Our code is optimized to run on intel compilers to assist in speed. The softwa re needed to run the code is as follows: Python (version >= 2.7.11) with python packages simplejson (version >=3.10.0), jsonschema (version >=2.5.1), genshi (version >=0.7), pint (version >=0.7.2), numpy (version 1.12.0), and scipy. The code also requires the intel compilers icc, icpc, and ifort. Based off extensive testing over the last few years, we estimate that a single run requires 10,000-30,000 time steps to find the optimum solution. From the previously mentioned approximate time for each step and the approximate number of steps needed for a given run, we estimate that each complete run requires roughly 5000 core-hours. That is, a single result requires ~2 weeks on a single node with 16 cores; this can be achieved by a sequence of several-day jobs to allow for machine downtime or other problems since we have the ability to restart jobs in the event that things break or go offline for maintenance. It is very straight forward and we only need to modify our driver file t o reference the directory of the stopped run. We can use however many cores are available on one node efficiently, we simply just adjust the number of cores that the run looks to utilize in our driver file. The code will use 100% of the cores of the node unless our driver file specifies the use of a subset of the cores available. This gives us the flexibility to utilize the full capacity of any node at Argonne (including the newer 32 node machines such as Beebop). We currently do not have the ability to run across nodes with MPI. The code could be adjusted to do so in the future if needed. Industry partnership: N/A Project URL: Requested allocation: 483840 Q1: 483840 Q2: 0 Q3: 0 Q4: 0 Justification: N/A Storage requirements: 1 TB should be fine. The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Request: BebopPerformance
by Low, John J. 28 Aug '17

28 Aug '17

Ray, An allocation or 100K core hours will not allow for much optimization or testing. John J. Low Principal Computational Science Specialist Computing, Environment and Life Sciences Building 240, 2143 9700 South Cass Avenue Argonne National Laboratory Argonne, IL 60439. 630-252-0045 www.linkedin.com/pub/john-low/15/8b0/5aa/ -----Original Message----- From: allocations-admins <allocations-admins-bounces(a)lcrc.anl.gov> on behalf of "Raymond A. Bair" <rbair(a)anl.gov> Reply-To: LCRC Allocations Admins <allocations-admins(a)lcrc.anl.gov> Date: Monday, August 28, 2017 at 12:42 PM To: LCRC Allocations Admins <allocations-admins(a)lcrc.anl.gov> Subject: Re: [allocations-admins] [LCRC Accounts] Project Request: BebopPerformance This looks like an ongoing project that needs to start now. So we can approve 100K for Q4. What process do we want to use to also have it included in the FY18 requests? Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 8/23/17, 11:21 AM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Dan Olson Applicant's institution: ANL Applicant's division: CELS Project Name: BebopPerformance Project title: System performance analysis/improvement with Intel/Cray Associated funding: Other Systems: Science: Project description: This will be used for collaborative work with Intel and Cray to improve the performance of the Bebop system. Industry partnership: Project URL: Requested allocation: 4000000 Q1: 1000000 Q2: 1000000 Q3: 1000000 Q4: 1000000 Justification: . Storage requirements: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

Re: [allocations-admins] [LCRC Accounts] Project Request: BebopPerformance
by Bair, Raymond A. 28 Aug '17

28 Aug '17

This looks like an ongoing project that needs to start now. So we can approve 100K for Q4. What process do we want to use to also have it included in the FY18 requests? Ray ----------------------------------------- Ray Bair Argonne National Laboratory and the University of Chicago On 8/23/17, 11:21 AM, "allocations-admins on behalf of accounts(a)lcrc.anl.gov" <allocations-admins-bounces(a)lcrc.anl.gov on behalf of accounts(a)lcrc.anl.gov> wrote: Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Dan Olson Applicant's institution: ANL Applicant's division: CELS Project Name: BebopPerformance Project title: System performance analysis/improvement with Intel/Cray Associated funding: Other Systems: Science: Project description: This will be used for collaborative work with Intel and Cray to improve the performance of the Bebop system. Industry partnership: Project URL: Requested allocation: 4000000 Q1: 1000000 Q2: 1000000 Q3: 1000000 Q4: 1000000 Justification: . Storage requirements: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list allocations-admins(a)lcrc.anl.gov https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Project Request: 3DDES
by accounts＠lcrc.anl.gov 28 Aug '17

28 Aug '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Ramesh Balakrishnan Applicant's institution: ANL Applicant's division: LCF Project Name: 3DDES Project title: Detached Eddy Simulation of Separated High Reynolds Number Flows Associated funding: ASCR Funding via ALCF Other Systems: Theta, Mira, Bebop Science: I am requesting 10M core hours, for the whole year, on Bebop, to develop detached eddy simulation models that can serve as a wall model for moderately separated flow in thick boundary layers. Examples of such flow are the atmospheric boundary layer (ABL) over complex terrain, large eddy breakup devices (LEBU), and separated flow past airfoils at high angles of attack (i.e. greater than 10 degrees). Project description: Preliminary simulations of the ABL over Complex Terrain have been carried out on Bebop during the Early User Period, where the PI had an opportunity to run and test out modified version(s) of OpenFOAM, SU2, and FDL3DI on Bebop. The simulations were run on meshes ranging from 28M grid points to 140M grid points, and on a maximum of 4096 cores of Bebop. The codes make use of MPI and can optionally use OpenMP. In addition, one can vectorize the code for the KNL nodes (in which case OpenMP is not needed at all). During the early user period, the vectorization and OpenMP options were not tested out. The flat MPI codes do scale well on both the KNL and Broadwell nodes, although the time-to-solution is about five to six times slower on the KNL nodes (when compared to the Broadwell nodes). This is something that I plan to investigate in more detail as part of this allocation request. Industry partnership: None at this time. Project URL: Requested allocation: 10000000 Q1: 2500000 Q2: 2500000 Q3: 2500000 Q4: 2500000 Justification: modified OpenFOAM: 35,000 finite volumes per core (on smallest number of cores) ======================================= cores time/time_step 4096 (KNL) 27 seconds (64 cores/node x 64 nodes) 2048 (KNL) 43 seconds (64 cores/node x 32 nodes) 1024 (KNL) 75 seconds (64 cores/node x 16 nodes) 4096 (BDW) 2 seconds (32 cores/node x 128 nodes) 2048 (BDW) 5 seconds (32 cores/node x 64 nodes) Please note that the 10M core hours (for the whole year) is requested on Bebop. 1024 (BDW) 12 seconds (32 cores/node x 32 nodes) modified SU2: 12,000 finite volumes per core (on smallest # of cores) ================================= solver cores time/time_step FV 1024 (KNL) 11 seconds FE_DG (p4) 1024 (KNL) 16 seconds (same # finite volumes/core) FV 2048 (KNL) 8 seconds FE_DG (p4) 2048 (KNL) 11 seconds (same # finite volumes per core) Storage requirements: 5TB I will be generating quite a few time slices of the simulation. The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for nano-soft-assembly
by support＠lcrc.anl.gov 26 Aug '17

26 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=nano-soft-ass…

1 0

New LCRC report submitted for nano-soft-assembly
by support＠lcrc.anl.gov 26 Aug '17

26 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=nano-soft-ass…

1 0

New LCRC report submitted for grafted_colloids
by support＠lcrc.anl.gov 24 Aug '17

24 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=grafted_collo…

1 0

New LCRC report submitted for grafted_colloids
by support＠lcrc.anl.gov 24 Aug '17

24 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=grafted_collo…

1 0

[LCRC Accounts] Yearly Allocation Request for NuclearStructure
by accounts＠lcrc.anl.gov 24 Aug '17

24 Aug '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Richard Chasman Project Name: NuclearStructure Division: PHY Project title: Optimization of many-body wavefunctions Associated funding: DOE/Office of Nuclear Physics Other Systems: desktop Science: The purpose of this work is to obtain a good description of low-lying nuclear states, going beyond a mean-field description. It becomes necessary to go beyond the mean-field approximation when the nucleus of interest is transitional in some sense, e.g. transitional between a spherical shape and a deformed shape. The need to go beyond a mean-field description is particularly important for excited states. Project description: We are now studying the low lying states of nuclei in which T=0 and T=1 pairing play an important role in addition to the deformation driving multipole-multipole interactions. We have extended our code to look at the anomalously large M1 matrix elements in nuclei with roughly equal numbers of protons and neutrons. We would like to extend our approach to larger systems.This requires faster cpu's and alot more memory. Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 10000 Q1: 2500 Q2: 2500 Q3: 2500 Q4: 2500 Justification: Storage requirements: 1 TB Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for NuclearStructure
by support＠lcrc.anl.gov 24 Aug '17

24 Aug '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=NuclearStruct…

1 0