Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Ramesh Balakrishnan Applicant's institution: ANL Applicant's division: ALCF Project Name: ALCF_BOEING_BLUES Project title: On the assessment of a proprietary unstructured finite volume solver for massive parallelization for hybrid RANS-LES computations Associated funding: Other Systems: Boeing Linux Clusters Science: Motivation: The Boeing BCFD code is an industrial-strength unstructured grid flow solver that is used in both the defense and commercial parts of Boeing to perform aerodynamic and propulsion analysis. It is seeing increased use for high-fidelity unsteady simulations that are compute intensive and additional scaling/performance is desired to facilitate analysis cycle time. The purpose of this proposal is to examine the performance of BCFD on a representative class of problems on its typical hardware to identify potential avenues for improving performance and porting to the Blue Gene/Q. This plan may lead to a follow-up effort to begin the modifications for efficient use of Blue Gene/Q hardware. Project description: Problem Description: An offset diffuser that was used in the first AIAA Propulsion Aerodynamics Workshop will be used as a representative geometry, as illustrated in Figure 1. Current results will be extended to add an additional level of grid refinement and a range of turbulence modeling techniques will be examined ranging from RANS to hybrid RANS-LES and, ultimately, to LES. This configuration provides a representative geometry that will provide accurate representation of industrial efforts and extend the state of the art to which LES has been performed. Computational Challenges: Offset diffusers often lead to local regions of separated flow that decrease engine performance and may lead to significant integration issues that are presently mitigated by increasing the length (and weight) of the diffuser. This often leads to increased size and cost of the overall vehicle. In order to reduce the impact of the separation, improved understanding of the detailed flow situation is desired. To this end, computational fluid dynamics (CFD) with high fidelity turbulence models can provide significant insight. However, these methods require intensive compute resources and lengthy times to obtain statistically meaningful data. This proposal will evaluate the accuracy and computational performance characteristics of a range of turbulence modeling approaches for this problem to provide better insight into both physical models and numerical algorithms. Recent (2013) modifications to the BCFD code have been made to improve its scaling on large -scale problems. A hybrid OpenMP/MPI method has been implemented that has significantly improved scaling as demonstrated in Figure 3. However, additional performance gains through algorithm modifications may still be possible, especially for application on the Blue Gene/Q system. Allocation Request: Meshes for the S-duct case appropriate for hybrid LES schemes are expected to be about 200 million cells (RANS fine grid from PAW was 60M cells). For unsteady simulations with reasonable throughputs, this would be computed on about 85-100 16-core compute nodes. At this level of grid size per processor, each physical time step will require about 150 wall-sec (15 cycles at 10 s/cycle). For a meaningful simulation, approximately 10K time steps will be required (0.50 sec at time steps of 50 microsec), leading to a cost per case of 416 wall-hours or 650K core-hours. With an expected 5 simulations required, this leads to a desired allocation of 1.5M core hours. Job Characterization: Typical cases 80-100 nodes (see above); Scaling cases could go to to approximately 150 nodes. Personnel: Ramesh Balakrishnan (Argonne National Lab): Perform analysis of code performance and develop plan for code modification for improved performance on Blue Gene/Q Mori Mani (Boeing): Develop analysis cases and assess both code scaling and physical modeling performance Mark Fisher (Boeing): Install BCFD on blues and assist in porting current code to Blue Gene. Provide guidance on current process implementation and concerns. Andrew Cary (Boeing): Provide user consulting on BCFD including appropriate case set-up for current practices. Provide insight into code algorithms and options. Project URL: Requested allocation: 1500000 Q1: 375000 Q2: 375000 Q3: 375000 Q4: 375000 Justification: I have a PDF that have this scaling information. I can attach it to an email that is generated by you system upon successful submission of this request. The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System