Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Jiali Wang Applicant's institution: ANL Applicant's division: EVS Project Name: Climate_Water Project title: Linking Climate to Water: Implementing a 4-km Regional Climate Model with Hydrologic Model Coupling (WRF-Hydro) using Argonne’s HPC Resources Associated funding: LDRD Other Systems: Science: This project will be obtained through the integration/development and testing at high-resolution of the linked WRF and WRF-Hydro system, optimizing the model performance on LCRC computing clusters and developing a streamlined input data generation capability and necessary post-processing software. The capability will provide a modeling system at very high resolution to: a) quantify the impact of the interactive feedbacks between hydrological, hydraulic systems with the atmospheric moisture dynamics; b) quantify the impact of enhanced convection resolving on surface hydrology and in particular stream flow rates over multiple events and years; and c) estimate the model uncertainty due to the sensitivity of model to multiple parameters. The objectives of this project are to (a) develop a comprehensive database of hydrologic and hydraulic flow parameters for the historical and future projections that correspond to our existing regional scale climate mode l database (Wang and Kotamarthi, 2015; Zobel et al. 2017); (b) initiate fully coupled WRF and WRF-Hydro modeling simulations at convective resolving scales (4 km or less grid spacing) for selected cases at spatial scales ranging from sub-regional (Midwest, Great Plains, etc.) to CONUS; and (c) link the WRF-hydro model to PEST to do model calibration and estimate model’s uncertainty due to the sensitivities of model to various parameters. This will create a hydrological database that leverages the large database of regional climate model product that we have built over the past few years and establishes the models’ machine readiness at LCRC. Project description: The proposed work will be performed as three major tasks. They are: (a) linking an independent tool (PEST) to WRF-Hydro to conduct calibration and uncertainty analysis automatically based on user-specified parameter range and selected parameter intervals. (b) running WRF-Hydro offline to explore the potential changes of streamflow in future climate, and (c) develop, enhance and implementation the coupled WRF-Hydro model at 4 km for the WRF on HPC clusters and perform simulations for select case studies and multi-year simulations over regions ranging from Midwest to the size of CONUS. Using nesting, we can turn on the execution of WRF-hydro in certain regions (e.g., Midwest and southern Great Plains), which shows significant performance improvement (Malakar et al. 2012). In the past several months, we have developed a parallel version of PEST which can be directly used on HPC such as LCRC computers. This tool bridge the WRF-Hydro model and the sensitivity and uncertainty analysis, which is essential in hydrological modeling practice. Tests on a smaller domain covering Midwest has been conducted on Blues. The general protocol will be used and tested for the simulations over entire CONUS. The offline WRF-Hydro simulations will use the 12km regional scale model output generated by our group over the past four years to provide the gridded meteorological input to the WRF-Hydro system. The output of the 12km WRF regional scale simulations includes six ensemble members of WRF simulations driven by three different GCMs considering various model setup (e.g., interior nudging technique, bias correction for the boundary conditions, etc.), and two different emission scenarios for the future periods. These three GCMs capture the range of responses of the CMIP5 models to greenhouse gas increase without doing every model in between. The future scenarios considered for the future period include Representative Concentration Pathways (RCP) 4.5 and RCP 8.5. A fully coupled atmosphere-hydrology model system, the WRF coupled with WRF-Hydro (WRF/WRF-Hydro), will be adopted. We propose to standardize the model for a 4-km resolution in the WRF model. At this resolution the convective processes in the atmosphere are fully resolved in WRF and we have shown that 4km resolution together with Morrison microphysics provide significant improvement in simulating precipitation over much of CONUS (see Figure 1) (Wang and Kotamarthi 2014; Chang et al. 2017). The model simulations will be performed at 4 km for the atmosphere and land component and 300 meters for the hydrological component. The development of high resolution terrain-routing grid is a time intensive process that requires the use of ArcGIS tools. We will investigate optimal data movement strategies between the WRF and WRF-hydro when they are coupled. These simulations will be followed by one-year simulation that covers the entire CONUS to establish the feasibility of using WRF-Hydr o for long-duration simulations for climate scale applications. Computational Performance of WRF-Hydro on Bebop: We will include surface or overland flow routing, subsurface routing, gridded-based channel routing, and lake/reservoir routing in the WRF-Hydro simulations. The hydrological output will be on a spatial resolution of 300 meters over the entire CONUS. Our experience with Bebop is, using 180 processors (5 nodes and 36 cores per node), running one-year simulation needs 80 wall-clock hours. So the core-hours for one-year simulation is 180*80=14400 First, we will work on model calibration using the HPC-version parallel-PEST tool we have developed in past several months. We plan to allow one-year spin-up time running the land surface module only to allow the soil feature to get equilibrium. We will then conduct one-year calibration considering many parameters that the WRF-Hydro is sensitive to. The number of runs for calibration will be at least 500, which will be automatically conducted by PEST. The exact number of runs will be decided by the validation statistics of PEST. The statistics is generally consider both spatial and temporal performance of the model compared to observation. Therefore the core-hours for the model calibration for one-year would be at least 14400*500=7.2 millions. These simulation can be conducted in parallel for each scenario that is important to the model’s performance. We hope that we can finish at least a portion of the calibration by the end of September. Second, we will apply the optimal parameters that we find from the first step to conduct the hydrological simulation for the entire CONUS. We plan to simulate a 10-year period (Year 1995-2004) for current and a 10-year periods for future considering two greenhouse gas concentration scenarios (e.g, RCP4.5 and RCP8.5). The meteorological input will be from our WRF simulation that we have completed in the past several years. We have done extensive model evaluation for all the ensemble members we have, so we are able to select one member that has relatively the best performance for precipitation and other variables. The core-hours would be 14400*10*3= 432000. These simulation will be conducted continuously for each 10-year period, and the three 10-year periods can be conducted in parallel. They will take 2-3 months in wall-clock time excluding the queue time. Third, a fully coupled atmosphere-hydrology model system, the WRF coupled with WRF-Hydro (WRF/WRF-Hydro), will be compiled on Bebop. The model simulations will be performed at 4 km for the atmosphere and land component and 300 meters for the hydrological component. These simulations will be followed by one-year simulation that covers the entire CONUS to establish the feasibility of using WRF-Hydro for long-duration simulations for climate scale applications. Our experience with Edison for running 4km WRF simulation, is that one-year simulation will take ~24 hours using 640 processors. Coupling hydrological modeling with the WRF simulation will cost 30% more time than running the WRF model (personal conversation with wrf-hydro team). Therefore, one-year simulation running WRF coupled with WRF-Hydro will take ~20,000 core-hours. Our tests show that Bebop has similar computational performance to Edison. Requested allocations: 1st QTR: 500K hours for model calibration and simulation (wrfhydro offline) 2nd QTR: 500K hours for model calibration and simulation (wrfhydro offline) 3rd QTR: 500K model simulation (wrfhydro offline and online testing) 4th QTR: 250K model simulation (wrfhydro online testing) Industry partnership: Project URL: Requested allocation: 1750000 Q1: 500000 Q2: 500000 Q3: 500000 Q4: 250000 Justification: The WRF-Hydro model considers surface or overland flow routing, subsurface routing, gridded-based channel routing, and lake/reservoir routing, among which the overland flow is the most time consuming while it is very important for describing the water flow on the surface especially for flood events. Conducting overland flow routing will slow down the simulation by a factor of two. The model can scale up to some extent, for example, using 20 nodes only increase simulation-hours by 40% compared to using 10 nodes, but because of load balance issues in the model design the model doesn’t scale up well. Our collaborators from ALCF are investigating into this and we expect to improve the model scaling problems over the next six months. We have contacted the primary developer of the WRF-Hydro at NCAR, Dr. Dave Gochis, and will establish a partnership with his team. The goal of this partnership will be to become co-developers of the model with a particular focus from the Argonne team on computational efficiencies, input and output (I/O) performance and coupling of additional process models (e.g., additional land surface model and ecology). Storage requirements: 1TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System