- Allocations-admins - mailman3.cels.anl.gov

Testing for ServiceNow 1
by Roberts, John E. 18 Sep '17

18 Sep '17

testing -- John Roberts HPC Systems Administrator

1 0

New LCRC report submitted for HEIGHTS-3D
by support＠lcrc.anl.gov 12 Sep '17

12 Sep '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=HEIGHTS-3D

1 0

[LCRC Accounts] Yearly Allocation Request for HEIGHTS-3D
by accounts＠lcrc.anl.gov 12 Sep '17

12 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Tatyana Sizyuk Project Name: HEIGHTS-3D Division: Project title: Advanced 3D modeling of Laser-Produced Plasma Processes for carbon nanostructures formation and EUV nanolithography in colliding plasmas and other advanced plasma applications. Modeling of transient events in full 3D fusion reactor environment. Associated funding: NSF, PIRE grant Other Systems: Science: Laser/target interactions: Laser Produced Plasma (LPP) systems are currently being considered as the main source of 13.5 nm photons for extreme ultraviolet lithography (EUVL) which is the technology for the manufacture of next generation computer chips. Development of compact high-resolution microscopy for water and carbon rich tissues such as proteins and lipids (Water-Window microscopy) is also based of LPP systems using various target configurations for plasma producing. The efficiency of these sources will determine the cost of future devices. Detailed source optimization requires significant experimental work and very costly efforts. Advanced modeling allows to accelerate the optimization process predicting optimum conditions and parameters based on fundamental analysis of all physical processes involved. Realistic predictions can only be done using well-benchmarked models with detail description of all the processes involved in complex multi-beam laser systems. Recently, the HEIGHTS package was used for modeling of colliding plasma experiments to study nanostructures formation in the chamber at the conditions of IFE experiments. The package was used also for modeling of EUV sources produced by multiple beams. These simulations are very time consuming and require 3D detailed modeling of several physics processes. Plasma/wall interactions in magnetic fusion reactor: A fundamental issue in tokamak operation related to power exhaust during plasma instabilities is the understanding of heat and particle transport from the core plasma into the scrape-off layer and to plasma-facing materials. During abnormal and disruptive operation in tokamaks, radiation transport processes play a critical role in divertor/edge-generated plasma dynamics and are very important in determining overall lifetimes of the divertor and nearby components. This is equivalent to or greater than the effect of the direct impact of escaped core plasma on the divertor plate. We have developed and implemented comprehensive enhanced physical and numerical models in the upgraded HEIGHTS package for simulating detailed photon and particle transport in the evolved edge plasma during various instabilities. Project description: The developed models and packages will be used for 1) the analysis of edge plasma and materials response during transient events in entire fusion reactor geometry and 2) the analysis and optimization of colliding plasmas in application to carbon nanostructures formation and EUV lithography. The developed models will be benchmarked with in-house experiments at our CMUXE center. We will upgrade the HEIGHTS package to include detailed models for the description of laser/matter interactions in short and ultra-short laser systems with various materials and for accurate modeling of target evolution and dynamics. Parallel implementation of models will be analyzed to improve the efficiency of 3D HEIGHTS package. We will study the following state-of-the art multi-phase multi-physics engineering applications: 1. HEIGHTS 3D LPP package will be used for modeling of colliding plasmas produced from carbon and mixed targets to study plasma characteristics in stagnation layers and to predict new structures formation in dependence on LPP multi-beam device parameters. 2. The package will be used for the analysis of new methods for EUV sources development, particularly, optimization of geometry and multiple laser parameters for producing the efficient EUV sources. 3. HEIGHTS 3D package for modeling of fusion reactor environment will be used for the analysis of power exhaust issues in various tokamaks. Improvement of the efficiency of parallel calculations will be one of the important tasks for this comprehensive package optimization. We depend highly on the use of LCRC cluster since our comprehensive packages could not run efficiently on other systems with insufficient computer resources. We would like to request 450,000 allocation hours to conduct various optimizations, benchmarking, and parametric studies of the HEIGHTS 3D packages for several projects during FY2018. ~10% of the allocation will be for non-parallel jobs Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 450000 Q1: 100000 Q2: 100000 Q3: 150000 Q4: 100000 Justification: Storage requirements: Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: networkbench
by accounts＠lcrc.anl.gov 11 Sep '17

11 Sep '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Sudheer Chunduri Applicant's institution: ANL Applicant's division: LCF Project Name: networkbench Project title: Performance analysis of the network Associated funding: Other Systems: Science: network design evaluation from scalability, variability perspectives. Project description: We wish to study the performance of the fabric using benchmarking and application scaling on Bebop. We also understand the factors the contribute to the run-to-run performance variability on the system, first, we need to assess whether the variability is indeed an issue with the network design used in Bebop. We wish to extensively understand the routing, topology and scheduling effects on the performance variability. Industry partnership: Project URL: Requested allocation: 80000 Q1: 20000 Q2: 20000 Q3: 20000 Q4: 20000 Justification: Storage requirements: 1 TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: NWChemEx
by accounts＠lcrc.anl.gov 11 Sep '17

11 Sep '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Ray Bair Applicant's institution: ANL Applicant's division: CLS Project Name: NWChemEx Project title: NWChemEx Development, Testing, and Evaluation Associated funding: DOE-SC ASCR ECP Other Systems: Cori, Theta, Titan Science: The NWChemEx Project in the Exascale Computing Program is redesigning NWChem and reimplementing a selected set of physical models for pre-exascale and exascale computing systems. This will provide the computational chemistry community with a software infrastructure that is scalable, flexible, and portable and will support a broad range of chemistry research on a broad range of computing systems. To guide this effort, the project is focused on two inter-related targeted science challenges relevant to the development of advanced biofuels: modeling the molecular processes underpinning the development of biomass feed stocks that can be grown on marginal lands and new catalysts for the efficient conversion of biomass-derived intermediates into biofuels and other bioproducts. Solution of these problems will enhance U.S. energy security by diversifying the energy supply chain. In addition to the science challenges targeted in the NWChemEx project, the software infrastructure provided by NWChemEx will enable it to be extended to address a number of other problems that are critical to the respective missions of DOE. These include modeling the properties of aqueous solutions; simulating the chemical processes in internal combustion engines; understanding the sequestration, transport and cycling of energy byproducts in the environment; developing a predictive science of synthes is; understanding the metabolism of plants and microbes; understanding the formation of atmospheric aerosols; developing new materials for solar conversion; predicting the environmental chemistry of the actinides, and many more. Project description: NWChem is a complex suite of programs that provides a broad range of modeling and simulation capabilities for describing the electronic structure and molecular dynamics of molecules. NWChem consists of approximately 4 million lines of code with three-quarters of the code being written by a domain-specific language module (the tensor contraction engine, TCE). Molecular Structure and Energetics: In this first (current) phase, NWChemEx will be redesigned for pre-exascale and exascale computers and new algorithms for high accuracy electronic structure methods will be implemented. These capabilities will provide a high fidelity description of the active site that can be rigorously coupled to lower fidelity models of the larger environment to ensure a proper in situ description of the phenomena of interest. A basic molecular dynamics capability will also be implemented. This project will include software development and debugging, algorithm performance evaluation at modest scale, and benchmarks that we use to compare algorithms and implementations across computers. We expect that most of the test runs on Bebop will be 1000-10,000 cores and less than 4 hours. Industry partnership: Project URL: Requested allocation: 900000 Q1: 225000 Q2: 225000 Q3: 225000 Q4: 225000 Justification: NWChem scales well on Blues and Bebop, and we expect NWChemEx to have even better scaling properties. Storage requirements: 4TB is needed to keep reference (integral) datasets used in algorithm evaluation. The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for SHARP-IF
by support＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=SHARP-IF

1 0

[LCRC Accounts] Yearly Allocation Request for SHARP-IF
by accounts＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Rui Hu Project Name: SHARP-IF Division: NE Project title: SHARP – Intermediate Fidelity Modeling Associated funding: DOE-Nuclear Energy Other Systems: Science: An intermediate fidelity (IF) thermal-hydraulics modeling method is being pursued for a fast-running, modest-fidelity, whole-core transient analyses capability. The approach is essential for design scoping and engineering analyses and could lead to improvements in the design of the new generations of reactors and to the reduction of uncertainties in safety analysis. The IF method relies on multi-scale T/H modeling hierarchy, RANS based turbulence modeling, coarse-grid CFD methodology, and advanced system modeling methods to represent the fluid flow and heat transfer physics. Project description: The RANS-based commercial CFD code STAR-CCM+ is incorporated into the SHARP-IF for a multi-scale analysis of Advanced Fast Reactors. STAR-CCM+ will be continuously used for design evaluations of reactor systems and experimental facilities. Notably, advanced sodium fast reactor design concept and the natural convection shutdown test facility that is currently being built at Argonne. This type of simulations is essential to assure the safety of the proposed design. Also, the post-test simulation results of the NSTF will be compared against to the experimental results as a validation study of the CFD code. An advanced reactor system analysis tool SAM is being developed at Argonne for advanced reactor applications based on the MOOSE framework. Nek5000 is a highly scalable, spectral element CFD code developed at ANL. It provides a high-fidelity DNS/LES thermo fluid simulation capability for coupled multi-scale multi-physics nuclear reactor applications. It is planned in FY18 to develop and verify the coupled SAM/ Nek5000 simulation capability, and perform demonstration calculations. At Argonne, the CFD code STAR-CCM+ are routinely applied to simulations that use more than 200 cores, and good parallel performance has been observed. As problem size increases, i.e. for finer meshes and for large pin bundles, larger numbers of cores would be utilized and based on data from the developers good parallel performance will continue to be expected. On the other hand, the system code SAM is based on the MOOSE framework. The parallelization will be provided by the underlying MOOSE framework that is built on PETSc. The system code usually relies on coarse representation of the system thus does not require significant amount of computing resources. Nek5000 is a highly scalable code, and its scaling performance has been demonstrated with extremely large simulations on Intrepid and Mira. Industry partnership: Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 200000 Q1: 40000 Q2: 40000 Q3: 60000 Q4: 60000 Justification: Storage requirements: 1 TB Thank You, The LCRC Accounts System

1 0

New LCRC report submitted for Nek5000
by support＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Full project report at https://accounts.lcrc.anl.gov/reporting/project/2017/?project=Nek5000

1 0

[LCRC Accounts] Project Request: Carotene
by accounts＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Anh Ngo Applicant's institution: ANL Applicant's division: MSD Project Name: Carotene Project title: Manipulation of Origin of Life Molecular Complexes Associated funding: BES Other Systems: Science: Among many organic molecular systems, β-carotene and chlorophyll-a are the key ingredients for light harvesting complexes and biological systems in nature. β-carotene takes multiple responsibilities in photosystems; as an antioxidant, and as a charge and energy transfer agent. It also functions as a protector of excessive light by adsorbing a region of the visible light spectrum not accessible by chlorophyll . Chlorophyll-a can be found in photosynthetic complexes in plant leaves like spinach, algae, and bacteria, and has an absorption spectrum for all the wavelengths of the visible light except that of the green light. Both molecular systems are also considered to be acting as molecular wires for charge transport. Although β-carotene and chlorophyll-a are vital for photosystems associated to the origin of life, a direct visualization of their detailed structures in the molecular complexes is still elusive. In this work, we systematically investigate conformations of individual β-carotene and chlorophyll-a using both ab initio calculations and molecular manipulation schemes with a low temperature scanning tunneling microscope, which enables identification of their intimate structures in the molecular complexes Project description: To provide insight into electronic properties of individual β-carotene and chlorophyll molecules, we propose to perform a first-principles study for individual β-carotene and chlorophyll molecules adsorbed on Au(111) surface. The following calculations will be performed: (a) The initial calculations will focus on finding the stable geometries of individual β-carotene and chlorophyll on Au(111) surface. The total energies and electronic structure of several oriented molecule structures on Au(111) surface will be investigated . This will help improve our understanding of physical properties of molecules adsorbed on GNR grown on Au(111) surface . These theoretical results will be compared with the available STM experiment results performed by Saw Hla group at CNM, Argonne national lab. (b) In our further investigations we will calculate the total force, density of states and charges states of the all carotene and chlorophyll molecules. We will be able to validate our theoretical results with the ongoing experimental efforts at CNM, Argonne. This work will plan to use the high-performance computing facilities at LCRC. The first-principles calculations will be carried out using the electronic-structure code VASP. The Blues/Bebop cluster is particularly suitable for this research because the studied systems are of large size, usually around 600-900 atoms. Structure optimizations of the system including a carotene molecule deposited on GNRs on Au(111) surface [M/Au (111)] with different magnetic configurations will be carried out. By comparing different energies we will conclude which configuration corresponding to the ground state energy. Once ground state structure is found we will go on to address the total force, density of states and charges states. Energy optimization of M/Au (111) (~800 atoms) using GGA+U takes approximately 43008 CPU hours (16*16*168 = 43008) and we plan roughly 17 of such calculations. For total force calculations. The calculation takes roughly 6915 CPU hours (16*6*72) we plan to run 37 of these calculations. So our total request is 16*43008 + 35*(16*6*72)= 930048 core hours. In summary, the total requested allocations for this project is 930048 CPU hours. Industry partnership: Project URL: Requested allocation: 930048 Q1: 250000 Q2: 250000 Q3: 250000 Q4: 180048 Justification: VASP performance on Blues is well established, we use the latest version maintained on Blues and Bebop. Storage requirements: 1 TB is sufficient The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Yearly Allocation Request for radix
by accounts＠lcrc.anl.gov 08 Sep '17

08 Sep '17

Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Halim Amer Project Name: radix Division: MCS Project title: Scalable Parallel System Software Associated funding: DoE core, SciDAC, NSF, FASTOS Other Systems: breadboard (unlimited) ALCF Mira/Cetus/Tukey (16,000,000 cpu hours) Science: Our mission is to develop the technologies required to dramatically increase the productivity of scientists developing applications for parallel supercomputers. The focus is fourfold: integration of parallel programming tools, reuse of parallel program components, development of scientific computing toolkits and portable libraries, and exploration of requirements of future parallel computers. Project description: This allocation will be used for optimizations to various systems software components: (1) programming models and runtime systems, (2) data I/O and file systems, (3) fault tolerance and data compression, (4) operating systems, (5) data analysis and visualization, and (6) deep learning. Primary goal of the project is to perform research on improving the core systems software on high-end computing platforms. The radix allocation covers a large number of cross-cutting projects involving subsets of researchers in the group. A few core projects are presented here, but some of the techniques used are shared between them and are not easily separable (e.g., MPI improvements or resilience improvements): 1. Fault Tolerance and Data Compression - Local memory management * Advanced SDC Detection through Data Monitoring + Adapt the prediction techniques used in the SZ lossy compressor for SDC detection * Checkpoint size reduction to improve checkpoint/restart performance and energy efficiency + Adapt FTI for using lossy compression of checkpoints * Impact of lossy compression on Checkpoint/restart + Investigate fundamentally and experimentally the relation between discretization/truncation error with lossy compression error * End-to-end Detection of SDCs in workflows featuring a lossy compression stage + Compare combination of SDC detectors and lossy compressors with respect to the accuracy of SDC detection. - Distributed memory management * Integration of checkpointing in workflow environments + Adapt DECAF and/or SWIFT with FTI and perform performance evaluation * MPI Fault Tolerance + Improvements to MPICH to deal with faults * Resilient-MPI and FTI Integration + Experimentation and evaluation of FTI coupled with resilient-MPI - Improvements to lossy data compression quality assessment through Z-checker * Optimize offline quality assessment to better characterize the * various lossy compression techniques * Speed up Z-checker with parallel assessment algorithms * Augment Z-checker with online visualization of the in-situ lossy compression results. 2. Developpement and optimization of OpenMP and threading runtimes - Improvements to our BOLT OpenMP runtime and exploring new OpenMP capabilities - Improvements to the Argobots threading framework - Study of the above runtimes under fat Xeon cores as well as slow cores such as KNL 3. Lightweight, topology-aware, and thread-safe communication: - Improvements to MPI+threads * MPICH improvements to play well with threads, including improved locking and lock-free data structures. * Communication progress improvements in the presence of threads. - Lightweight communication * Low instruction-count and low-overhead irregular data movement. - Topology-aware communication * Optimizations to MPI virtual topology functionality 4. Data I/O, RMA and Active Messages: - Dynamic Locality-Aware Workflows * Dynamically distribute the tasks among nodes using Asynchronous Dynamic Load Balancer - Development and Evaluation of Composable Data Services * Exploration of lightweight, customized, domain-specific storage services to improve the performance of data-intensive HPC applications - High Performance Parallel I/O * The ROMIO MPI-IO implementation provides the foundation for many components of the HPC I/O stack. Continue tuning, debugging, and research efforts. 5. Data analysis and visualization: Testing and benchmarking of improvements to data movement infrastructure such as DIY and Decaf and to data model software such as MFA to prepare for next-generation machine architectures and to apply those improvements to applications across the lab that use those software platforms, such as cosmology, climate science, and materials science. The following areas will be tested on Blues and Bebop: - Relaxed BSP synchronization (e.g., for iterative algorithms, to overlap communication with computation and support irregular communication/computation patterns) * irregular communication patterns - Load-balancing algorithms (work stealing, adaptive decompositions) for unstructured time-varying mesh partitions (e.g., reactor design) * work stealing algorithms * adaptive (e.g. k-d tree) domain decompositions - Workflow management * User interfaces * Scalability, performance * Integration with distributed area workflows and system software * Application to HEP, climate, superconductivity, materials science - Data models * Multivariate functional approximation with adaptive knot insertion * Linear and nonlinear solutions * Domain segmentation 6. I/O enhancements for scalable deep learning - In FY17, we focused on optimizing the usage of mmap for deep learning. In FY18, we plan to move our focus on replacing mmap with direct I/O mechanisms. We will investigate mechanisms to use MPI I/O for direct I/O in an effort to improve performance through bulk I/O while maintaining correctness through fall-back mmap buffers. - Our work in FY17 focused on the Caffe framework. We plan to extend our work to newer deep learning frameworks such as Tensorflow and Caffe2. - We would like to investigate the impact of Intel Xeon Phi architecture on deep learning frameworks, and in particular how the improved computational resources would affect I/O performance. Industry partnership: Project URL: http://www.mcs.anl.gov/group/pmrs/ Current FY Hours Used: undetermined amount New FY Requested allocation: 5200000 Q1: 1300000 Q2: 1300000 Q3: 1300000 Q4: 1300000 Justification: 5.2M core hours requested, most of which would be using Bebop: We have six core projects. All projects have been demonstrated at scale in the previous years and form the core of the system software stack used in production on the Fusion and Blues machines. We expect to run around ~20 studies for each project on Bebop and on Blues, scaling from 1 node up to 512 nodes (average runtime of 1 hour) - Bebop (Broadwell): 6 (projects) x 10 (studies) x 1024 (1 to 512 nodes) x 36 (cores) x 1 (hour) = 2.2M - Bebop (KNL): 6 (projects) x 10 (studies) x 512 (1 to 256 nodes) x 64 (cores) x 1 (hour) = 2M - Blues: 6 (projects) x 20 (studies) x 512 (1 to 256 nodes) x 16 (cores) x 1 (hour) = 1M A small fraction of this would be used for debugging. Storage requirements: Persistent storage of the order of 100TB is required for our deep learning project. This project requires large datasets that take more than two weeks to generate. Recreating them every time on scratch space would be prohibitive; thus, the request for large storage space. Thank You, The LCRC Accounts System

1 0