Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Timothy J. Tautges Project Name: Fathom Division: MCS Project title: Fathom performance engineering Associated funding: DOE/NE, DOE/ASCR, DOE/BER Other Systems: (making proposals to Intrepid, Surveyor) Science: There are two primary areas of science goals for this project. For nuclear reactor simulation, we seek to construct very large mesh models, in the range of 50M-200M elements. These meshes will represent whole-core reactor models, and will support both thermal-fluid (Nek) and neutron transport (UNIC) simulations. The specific science goals for these codes include: resolving mixing behavior in both thermal and fast reactor cores evaluating the accuracy versus computational time for coupled thermal fluid / neutronics computations evaluating the accuracy of heterogeneous material neutron transport modeling, compared to lumped parameter models used in other codes support for various OECD benchmark participation, including the Vettenfahl T-junction and the Mitas benchmarks For climate modeling, we will continue developing the ParCAL library, based on MOAB; this effort seeks to parallelize the NCL climate data analysis tool. Our efforts this year will be to continue filling out the capabilities in the original NCL tool which are performed in ParCAL/MOAB, and demonstrating these capabilities on real data analysis efforts. Project description: MOAB is a library for representing, querying, and modifying mesh and associated data. MOAB supports the finite element zoo, plus polygons and polyhedra, and, more recently, includes an interface for structured mesh. MOAB uses an efficient array-based storage model, allowing applications direct access to memory storage for field data and, soon, to connectivity and coordinate data. MOAB uses an MPI model for parallel mesh, where mesh models appear locally as a serial mesh, with additional data available about parallel constructs such as inter-processor mesh interfaces and ghost layers. The primary means for accessing MOAB's parallel representation are to read/initialize and write mesh from/to a file. MOAB uses parallel HDF5 for parallel I/O of unstructured meshes, and parallel netcdf for structured mesh parallel I/O. Substantial work has been done to optimize this file I/O, as well as the resolution of shared inter-processor interface mesh and ghost layer exchange. Parallel I/O has been demonstrated on unstructured hex and tet meshes up to 64M elements, on up to 64k processors of Intrepid. Various performance issues are still being addressed, including enhancements to MPI-IO for single-file parallel I/O (in cooperation with Rob Latham at ANL). MOAB targets a variety of applications. For nuclear energy applications, the Nek CFD code and the UNIC neutronics code have been modified to read mesh from MOAB. Nek has been modified to export its solution variables through MOAB as well. The Nuclear Energy co-design project CESAR plans to make extensive use of MOAB, for mesh-based interactions as well as solution transfer between meshes. MOAB supports several climate applications, including ice sheet modeling and climate data visualization and analysis. MOAB will also be involved in at least one SciDAC application project, involving fusion edge modeling. For reactor simulation, we have also developed the Reactor Geometry (and mesh) Generator (RGG). This tool begins in the "assygen" stage by generating geometric and mesh models for a number of assembly types, then, in the "coregen" stage, copy/moves/merges these assemblies into an overall core lattice, with some assemblies copied many times. We have demonstrated good parallel speedups of the coregen stage of RGG, in some cases obtaining super-linear speedups due to decreased swapping and thrashing of virtual memory. This year's technical work will focus on: - testing and analyzing parallel I/O performance reading reactor simulation and climate data - testing and analyzing performance of parallel mesh-to-mesh transfer functionality in MOAB - performing parallel coupled multi-physics simulations for the NEAMS program Project URL: http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB Current FY Hours Used: undetermined amount New FY Requested allocation: 100000 Q1: 25000 Q2: 25000 Q3: 25000 Q4: 25000 Justification: MOAB has demonstrated parallel I/O out to 1024 cores on fusion, and 64k cores on Intrepid. For example, MOAB can read a structured mesh and a single solution variable, for a mesh of approximately 160M cells, in about ten seconds on 1024 processors, and less than 3 seconds on 64 processors. For unstructured meshes, MOAB reads meshes up to 64M elements in less than a hundred seconds on Intrepid. Execution times in both these cases seems to level off above about 128 processors, and we are working with Rob Latham of ALCF to understand why this happens. RGG demonstrates good scaling out to the total number of assemblies in a given core model, usually 100-300 for whole-core models. In the case of RGG, we have demonstrated super-linear speedups in some cases, due to fitting the entire model in memory that eliminates the need for virtual memory swapping (and thrashing, for large models). Thank You, The LCRC Accounts System