Hello, A yearly allocation for the LCRC cluster has been requested with the following updated information: Submitter/PI: Misun Min Project Name: Electromagnetics Division: MCS Project title: Computational Electromagnetics Associated funding: DOE AMR program and LDRD strategic initiative Other Systems: Argonne BG/P (Intrepid), Jaguar (OLCF) Science: A suite of electromagnetics codes has been under development at MCS/ANL that is based on unstructured hexahedral grids for high accuracy and scalable parallel performance to enable tera- and petascale simulation. The production code NekCEM, a high-order electromagnetic solver that we have developed under DOE projects, is currently capable of scalable simulations up to >131K cores on the Argonne BG/P for solving transport phenomena, including beam dynamics for accelerator devices and light transmissions for nanophotonics devices. To this end, we propose to dramatically increase the capabilities of NekCEM, producing a single, comprehensive, high-fidelity electromagnetic software package that runs in parallel beyond petascale for predictive modeling of nanosensors and photovoltaic solar cells, which will help researchers to discover and explore innovative solutions for current energy shortage, global warming, medical diagnoses, and national security. Project description: The breakthrough goal of this project is to develop NekCEM as an extreme-scalable computational tool for understanding the fundamental physics and predicting optimal designs of next-generation electromagnetic devices. Algorithmic development will include extreme-scale programming using MPI/SMP models; efficient time-stepping algorithms; efficient parallel communication kernels, I/O, visualization, performance enhancement; realistic simulations; and validation of computational results with experimental measurements. We consider a high-order SEDG method with a motivation summarized as follows: fewer grid points per wavelength required to achieve a given accuracy; efficiency, with CPU time depending on the degrees of freedom, not the order of approximation; and geometric flexibility with body-conforming meshes. NekCEM is currently capable of scalable simulations up to >131K cores on leadership-class machines such as the Argonne BG/P. The CPU time per time step is about 0.13 seconds on 131K cores for the case of 1.1 billion grid points, with the scalability achieving 75% efficiency on 131K cores for the number of grid points per core n/P=8,530 and 58% efficiency even with an unrealistically small number of n/P=224. Several I/O approaches have been implemented and extensively tested for the write-performance based on a collective I/O and reduced-blocking I/O for single and multiple outputs on up to 65K cores on BG/P in comparison with one POSIX file per processor (1PFPP). The best I/O rate was achieved from the reduced-blocking I/O with approximately 5 seconds to write a single snapshot of a billion-point simulation. Future tests include runs on OLCF Jaguar and upcoming BG/Q. We expect 4-5 project members. Project URL: Current FY Hours Used: undetermined amount New FY Requested allocation: 40000 Q1: 5000 Q2: 5000 Q3: 15000 Q4: 15000 Justification: Thank You, The LCRC Accounts System