Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Timothy J. Tautges Applicant's institution: ANL Applicant's division: MCS Project Name: Fathom Project title: Fathom performance engineering Associated funding: DOE/NE, DOE/ASCR, DOE/BER Other Systems: (making proposals to Intrepid, Surveyor) Science: There are two primary areas of science goals for this project. For nuclear reactor simulation, we seek to construct very large mesh models, in the range of 50M-200M elements. These meshes will represent whole-core reactor models, and will support both thermal-fluid (Nek) and neutron transport (UNIC) simulations. The specific science goals for these codes include: resolving mixing behavior in both thermal and fast reactor cores evaluating the accuracy versus computational time for coupled thermal fluid / neutronics computations evaluating the accuracy of heterogeneous material neutron transport modeling, compared to lumped parameter models used in other codes support for various OECD benchmark participation, including the Vettenfahl T-junction and the Mitas benchmarks For climate modeling, we will continue developing the ParCAL library, based on MOAB; this effort seeks to parallelize the NCL climate data analysis tool. Our efforts this year will be to continue filling out the capabilities in the original NCL tool which are performed in ParCAL/MOAB, and demonstrating these capabilities on real data analysis efforts. Project description: MOAB is a library for representing, querying, and modifying mesh and associated data. MOAB supports the finite element zoo, plus polygons and polyhedra, and, more recently, includes an interface for structured mesh. MOAB uses an efficient array-based storage model, allowing applications direct access to memory storage for field data and, soon, to connectivity and coordinate data. MOAB uses an MPI model for parallel mesh, where mesh models appear locally as a serial mesh, with additional data available about parallel constructs such as inter-processor mesh interfaces and ghost layers. MOAB's parallel capabilities have been the focus of work over the last year. The primary means for accessing MOAB's parallel representation are to read/initialize and write mesh from/to a file. MOAB uses parallel HDF5 for parallel I/O of unstructured meshes, and parallel netcdf for structured mesh parallel I/O. Substantial work has been done to optimize this file I/O, as well as the resolution of shared inter-processor interface mesh and ghost layer exchange. Parallel I/O has been demonstrated on unstructured hex and tet meshes up to 64M elements, on up to 64k processors of Intrepid. Various performance issues are still being addressed, including enhancements to MPI-IO for single-file parallel I/O (in cooperation with Rob Latham at ANL). MOAB targets a variety of applications. For nuclear energy applications, the Nek CFD code and the UNIC neutronics code have been modified to read mesh from MOAB. Nek has been modified to export its solution variables through MOAB as well. The Nuclear Energy co-design project CESAR plans to make extensive use of MOAB, for mesh-based interactions as well as solution transfer between meshes. MOAB supports several climate applications, including ice sheet modeling and climate data visualization and analysis. MOAB will also be involved in at least one SciDAC application project, involving fusion edge modeling. For reactor simulation, we have also developed the Reactor Geometry (& mesh) Generator (RGG). This tool begins in the “assygen” stage by generating geometric and mesh models for a number of assembly types, then, in the “coregen” stage, copy/moves/merges these assemblies into an overall core lattice, with some assemblies copied many times. We have demonstrated good parallel speedups of the coregen stage of RGG, in some cases obtaining super-linear speedups due to decreased swapping and thrashing of virtual memory. This year's technical work will focus on: generating mesh models for nuclear reactor simulation, using RGG performing climate data visualization and analysis as part of our “Parviz” project (PI: Rob Jacob, ANL) improving scalability of RGG improving scalability of MOAB using parallel I/O through parallel netcdf and parallel HDF5 In addition, this year we plan to submit a paper to the IPDPS conference describing our parallel I/O work, and a proposal to the 2012 ALCC call for 5M hours of compute time to support this same technical work. A draft version of the IPDPS paper will be available for review by September 15. The compute time requested in this call will be used for final timing data for the IPDPS paper and to support our application to ALCC, as well as to support the applications described above. Project URL: Requested allocation: 1000000 Q1: 250000 Q2: 250000 Q3: 250000 Q4: 250000 Justification: MOAB has demonstrated parallel I/O out to 1024 cores on fusion, and 64k cores on Intrepid. For example, MOAB can read a structured mesh and a single solution variable, for a mesh of approximately 160M cells, in about ten seconds on 1024 processors, and less than 3 seconds on 64 processors. For unstructured meshes, MOAB reads meshes up to 64M elements in less than a hundred seconds on Intrepid. Execution times in both these cases seems to level off above about 128 processors, and we are working with Rob Latham of ALCF to understand why this happens. RGG demonstrates good scaling out to the total number of assemblies in a given core model, usually 100-300 for whole-core models. In the case of RGG, we have demonstrated super-linear speedups in some cases, due to fitting the entire model in memory that eliminates the need for virtual memory swapping (and thrashing, for large models). The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System