January 2015 - Allocations-admins - mailman3.cels.anl.gov

Fwd: Project Request: DMDII-Pilot1-turbine
by Bair, Raymond A. 30 Jan '15

30 Jan '15

Sent from my iPhone Begin forwarded message: From: Paul Domagala <domagala(a)anl.gov<mailto:[email protected]>> Date: January 30, 2015 at 3:01:14 PM CST To: Raymond Bair <rbair(a)anl.gov<mailto:[email protected]>> Subject: Re: Project Request: DMDII-Pilot1-turbine Ray, The technical team discussed this on our last call. If we leave out any proprietary info, the underlying computation science can be published. Yes. Paul Paul Domagala Argonne National Laboratory On Jan 30, 2015, at 2:33 PM, Bair, Raymond A. <rbair(a)anl.gov<mailto:[email protected]>> wrote: Paul, Will the results of this work be published in the open literature, or is it all proprietary? Regards, Ray

1 0

Re: [allocations-admins] [LCRC Accounts] Project Request: DMDII-Pilot1-turbine
by Bair, Raymond A. 30 Jan '15

30 Jan '15

I queried Paul to see if this is proprietary work, or not. Ray ----------------------------------------- Ray Bair Computing, Environment, and Life Sciences Argonne National Laboratory and the University of Chicago TCS Building 240, Room 4122 9700 South Cass Avenue Argonne, IL 60439 email: rbair(at)anl.gov Phone: (630)252-5751 On 1/30/15, 2:29 PM, "accounts(a)lcrc.anl.gov" <accounts(a)lcrc.anl.gov> wrote: >Hello, > >A new project on the LCRC cluster has been requested. Please forward >the information on to the LCRC Allocation sub-committee. > > Applicant's name: Paul Domagala >Applicant's institution: ANL > Applicant's division: PSE > Project Name: DMDII-Pilot1-turbine > Project title: DMDII-sponsored S&M for Illinois wind turbine >mfgr > Associated funding: Direct funded by the Illinois Manufacturing >Laboratory (IML) under UI Labs > Other Systems: > Science: This pilot project will assess the >effectiveness of a computationally-based design process. Opportunities >may also exist to assess the scalability of design codes and commercial >M&S codes such as ANSYS CFX or Fluent. > >Scientific paper(s) may be submitted to peer-reviewed journals if the >effort results in unique computational or mathematical approaches or >methods. > Project description: ?Andreas Vlahinos, a computational modeling and >simulation expert under contract to UI Labs will execute this project in >conjunction with CLS & MCS scientists. > >Small Vertical Wind Turbine Performance >---------------------------------------------- >Conduct computational fluid mechanic (CFD) analyses of vertical wind >turbine designs to assess performance under a number of operating >conditions. > >Starting with manufacturer-supplied engineering models and operational >parameters, formulate the computational model and conduct detailed >simulations using CFD software of the contractor¹s choice. > >Work with Argonne staff to automate workflow and conduct parametric >design studies. Work iteratively with the manufacturer to optimize the >design. > Industry partnership: This project will be conducted in cooperation >with the Illinois Manufacturing Laboratory and the Digital Manufacturing >and Design Innovation Institute (DMDII). > >- The customer is an Illinois-based manufacturers. The identity of the >company and complete problem definition have not yet been released by the >DMDII Executive Board. > >- A CRADA is in process for this work. > Project URL: > Requested allocation: 170000 > Q1: 0 > Q2: 60000 > Q3: 90000 > Q4: 20000 > Justification: > Storage requirements: 1 TB > > >The requester has used undetermined amount hours of their initial startup >project. > >In addition to approving an initial amount, please specify a Category >and Subcategory for this project. For a list of the current selection >of approved categories, please see: > > https://wiki.lcrc.anl.gov/wiki/Processes/Categories > >Once the Allocation committee has approved the project, please go to >the Project Management page to create it: > >https://accounts.lcrc.anl.gov/projects.php > >Thank You, >The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: DMDII-Pilot2-grain
by accounts＠lcrc.anl.gov 30 Jan '15

30 Jan '15

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Paul Domagala Applicant's institution: ANL Applicant's division: PSE Project Name: DMDII-Pilot2-grain Project title: DMDII-sponsored S&M for Illinois grain transport DMDII-sponsored S&M for Illinois wind turbine mfgr Associated funding: Direct funded by the Illinois Manufacturing Laboratory (IML) under UI Labs Other Systems: Science: This pilot project will assess the effectiveness of a computationally-based design process. Opportunities may also exist to assess the scalability of design codes and commercial M&S codes such as LS-Dyna. Scientific paper(s) may be submitted to peer-reviewed journals if the effort results in unique computational or mathematical approaches or methods. Project description: Andreas Vlahinos, a computational modeling and simulation expert under contract to UI Labs will execute this project in conjunction with CLS & MCS scientists. Grain Transportation and Delivery System ---------------------------------------------- Formulate computational models of grain flow in commercial transportation delivery systems such as barges, grain elevators, rail cars and associated grain- moving systems. Investigators will evaluate computational methods/software such as mesh-free Lagrangian approaches (e.g. smoothed-particle hydrodynamics (SPH)) to determine the best approach for this class of problem. Andreas will work with Argonne staff to automate workflow and assess derivative-free optimization techniques. Conduct parametric design studies. The team will work iteratively with the manufacturer to optimize the design. Industry partnership: This project will be conducted in cooperation with the Illinois Manufacturing Laboratory and the Digital Manufacturing and Design Innovation Institute (DMDII). - The customer is an Illinois-based manufacturers. The identity of the company and complete problem definition have not yet been released by the DMDII Executive Board. - A CRADA is in process for this work. Project URL: Requested allocation: 170000 Q1: 0 Q2: 60000 Q3: 90000 Q4: 20000 Justification: Storage requirements: 1 TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Request: DMDII-Pilot1-turbine
by accounts＠lcrc.anl.gov 30 Jan '15

30 Jan '15

Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Paul Domagala Applicant's institution: ANL Applicant's division: PSE Project Name: DMDII-Pilot1-turbine Project title: DMDII-sponsored S&M for Illinois wind turbine mfgr Associated funding: Direct funded by the Illinois Manufacturing Laboratory (IML) under UI Labs Other Systems: Science: This pilot project will assess the effectiveness of a computationally-based design process. Opportunities may also exist to assess the scalability of design codes and commercial M&S codes such as ANSYS CFX or Fluent. Scientific paper(s) may be submitted to peer-reviewed journals if the effort results in unique computational or mathematical approaches or methods. Project description: Andreas Vlahinos, a computational modeling and simulation expert under contract to UI Labs will execute this project in conjunction with CLS & MCS scientists. Small Vertical Wind Turbine Performance ---------------------------------------------- Conduct computational fluid mechanic (CFD) analyses of vertical wind turbine designs to assess performance under a number of operating conditions. Starting with manufacturer-supplied engineering models and operational parameters, formulate the computational model and conduct detailed simulations using CFD software of the contractor’s choice. Work with Argonne staff to automate workflow and conduct parametric design studies. Work iteratively with the manufacturer to optimize the design. Industry partnership: This project will be conducted in cooperation with the Illinois Manufacturing Laboratory and the Digital Manufacturing and Design Innovation Institute (DMDII). - The customer is an Illinois-based manufacturers. The identity of the company and complete problem definition have not yet been released by the DMDII Executive Board. - A CRADA is in process for this work. Project URL: Requested allocation: 170000 Q1: 0 Q2: 60000 Q3: 90000 Q4: 20000 Justification: Storage requirements: 1 TB The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System

1 0

Re: [allocations-admins] [LCRC Accounts] Project Allocation Request
by John Valdes 28 Jan '15

28 Jan '15

OK for 50K. On Mon, Jan 26, 2015 at 11:45:00AM -0600, accounts(a)lcrc.anl.gov wrote: > Hello, > > A change in allocation has been requested: > > Requester: dduke (Daniel Duke) > Project: XRayFuelSpray > Title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics > Description: The cavitation modeling component of the project is being performed using the OpenFOAM framework. A number of state-of-the-art numerical solvers have been implemented in OpenFOAM; namely incompressible-liquid and fully compressible homogeneous relaxation cavitation models and more recently a fully compressible homogeneous relaxation cavitation model which includes non-condensable gas modeling (HRMFoam). These codes were developed by Schmidt et al at the University of Massachusetts-Amherst, and we continue to collaborate with them on code development and efficiency improvements. > > In the last year, we have made some substantial improvements to the cavitation model and the parallel efficiency of its implementation on Blues/Fusion. With the support of LCRC technical staff and our collaborators at the University of Massachusetts-Amherst, we have implemented a custom build of OpenFOAM-3.0 on Blues against the system native MPI, and we have made the necessary changes to the HRMFoam solver???s source code in order to move from the older OpenFOAM-1.6 to the new 3.0 framework. This has led to substantial gains in parallel performance; up to 20% on 32 cores and 12% on 64 cores on Blues. A scalability study has been performed with the new code, and details are given in the Large Allocation Efficiency statement. > In the new OpenFOAM-3.0 build, we have successfully implemented both Large Eddy Simulation (LES) and Realizable k-epsilon (KE) turbulence models. Furthermore, we have implemented code which allows us to take virtual projections of the three-dimensional simulations so that they may be quantitatively compared with the x-ray experiments. > > In the following year, we plan to extend high-fidelity cavitation simulations by undertaking longer simulation times in order to capture lower-frequency hydrodynamic phenomena which we have observed in our experiments. Improvements in efficiency will allows us to pursue longer run-time simulations. We also intend to simulate the effects of varying pressure and flow rate on cavitation, in order to compare against new x-ray fluorescence data from APS. > The gas jet simulations which we plan to undertake in Q1 are performed with the CONVERGE CFD software, its use is already well established at LCRC. > > Tomographic reconstruction of gasoline direct injection spray measurements from APS will be conducted using the TomoPy software package developed at APS. We intend to run single-node jobs to efficiently perform multiple reconstructions of large (multi-GB) datasets. We have also successfully compiled a Paraview server on our Fusion dedicated nodes which allows us to rapidly visualize the results of our simulations and reconstructions. Using Blues and Fusion will allow us to achieve rapid turnaround of results during beamtime. This will be a significant aid to the experimental program. As the quantity of tomography work increases, we may consider purchasing more dedicated nodes. > Current: undetermined amount > Justification: Our requested allocation for next year will exceed 0.5M core-hours in order to encompass the gas jet simulations in Q1 and tomographic reconstruction work in Q2-4. > > The majority of core hours will still be dedicated to OpenFOAM simulations, which will be undertaken on both dedicated Fusion nodes and also on Blues, using OpenFOAM-3.0. We have demonstrated notable improvements in scalability on Blues as detailed above. > > A scalability study of OpenFOAM-3.0 performance with multi-million cell meshes has been undertaken, demonstrating excellent linear scalability. We observe 81% efficiency relative to single-CPU performance, scaling linearly up to 320 cores (20 nodes) with a 15-million cell mesh (a typical upper limit). We have implemented Scotch decomposition in order to ensure good load balancing & minimization of cross-processor communication boundaries. The present OpenFOAM simulations use static meshes, so load rebalancing at runtime is not a concern. We intend to run typical job sizes of 64 to 128 cores; smaller than the maximum job size for which good scalability has been demonstrated. Further details regarding scalability studies can be obtained by contacting the project PIs. > > Requested: 50000 > > A specific reason has been given: > We have had to accelerate our schedule for completing OpenFOAM CFD calculations this year in order to meet a number of very early conference paper deadlines. As such, we have already exhausted our allocation rather early. We would like to request an addition 50k hours this quarter, to complete the final simulations in order to meet our publication deadline. If you have any questions or require further details please contact PI Duke (dduke(a)anl.gov) > > This needs to be approved and the final allocation amount decided upon. > > Thank You, > The LCRC Accounts System > _______________________________________________ > allocations-admins mailing list > allocations-admins(a)lcrc.anl.gov > https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 28 Jan '15

28 Jan '15

Hello, A change in allocation has been requested: Requester: zmei (Zhigang Mei) Project: Multilayer_nitride Title: Atomic-scale design of radiation-tolerant multilayer coatings for nuclear applications Description: We use atomistic MD simulations to investigate one of the interface-governed behaviors exhibited by multilayered nanocomposite of ZrN and AlN, i.e., their substantially enhanced resistance to radiation damage compare to pure ZrN or AlN. Results from MD simulation will be compared to experimental ion irradiation studies of multilayered AlN/TiN nanocomposite [Milsosavljevic, J. Phys. D: Appl. Phys. 43 (2010) 65302]. The interatomic potentials for the nitrides are based on the modified embedded atom method (MEAM). However, there is no ternary potential available for ZrN-AlN. We will use the so-called force matching method as implemented in potfit [Brommer, Modelling Simul Mater Sci Eng 15 (2007) 295] to develop such ternary potential. To build a potential suitable for the simulation of a complicated ternary system in a wide ranges of pressure and temperature, we have to prepare an extensive set of reference configurations representing all kinds of interesting stru ctures. Each configuration is a small structure model corresponding to one of the possible states of the ZrN-AlN system. VASP will be used to calculate energy, force and stress of the reference configurations with atoms up to 250. The total computational time for DFT calculations will be about 30000 CPU-hours. Model ZrN-AlN bilayers are constructed in the experimentally observed orientation relation [Milosavljevic, J Phys. D: Appl. Phys. 43 (2010) 065302], in which a ZrN {111} plan neighbors an AlN {0001} plane and an interfacial ZrN <112> direction lies parallel to an AlN <2110> direction. The constituent ZrN and AlN layers are 4.5 nm thick, have periodic boundaries in the interface plane, and terminate in free surfaces away from the interfaces. The ZrN/AlN multilayer nanocomposite can be created in a similar way by stacking 4.5 nm-thick ZrN and AlN crystalline slabs, in which each ZrN or AlN layer contains up to 100000 atoms (22.5×22.5×4.5nm). The layer thickness, number of layers and interface orientation can be controlled in deposition experiments, which will be investigated independently by atomistic simulations. The ZrN-AlN+ZBL potential is used to model collision cascades in in cubic ZrN, hexagonal AlN, and in the vicinity of ZrN-AlN interfaces formed in the multilayer composites. Cascades are initiated by giving a selected atom-the primary knock-on atom (PKA)-a large kinetic energy. The Kinetic energies given to PKAs are chosen to be representative of Ar+ implantation experiments carried out on ZrN-AlN nanocomposites, where the energy of the implanted ions was 200 keV. In the multilayer model systems PKAs are selected from the central layer. For the collision cascade simulations, the first few pico-seconds require extreme fine time step to capture the recombination of point defects, and then nano-second time scale simulations are necessary to reach a thermal equilibrium for the defect structure. By combining all the factors to be studied, we will expect about 25 production jobs. Our initial test shows that a single job needs about 25000 CPU-hours (256 CPU×96 hours). We therefore r equest the total computational time of 650000 CPU-hours. Current: undetermined amount Justification: The used molecular dynamics code LAMMPS has been demonstrated to be cable of simulating millions of atoms using thousands of CPU efficiently. The detailed benchmarks of the parallel efficiency of lammps on different platform can be found from the website of the author http://lammps.sandia.gov/bench.html. With the typical number of CPU (256 CPU) in our MD simulations, the parallel efficiency of LAMMPS on Blues will be above 70%. Requested: 150000 A specific reason has been given: The requested computational time was fully allocated as requested. Meanwhile, DFT calculation for random alloy structures is much more demanding than expected. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 28 Jan '15

28 Jan '15

Hello, A change in allocation has been requested: Requester: ssom (Sibendu Som) Project: LES-sprays Title: LES modeling for diesel spray and engine simulations Description: The project is divided into four major tasks to be performed in 3 years in collaboration with Convergent Science Inc., and Cummins Engine Company: Task 1: An one equation non-viscosity dynamic structure model large eddy simulation (LES) based turbulence modeling approach will be implemented in CONVERGE. Grid-convergence on critical spray parameters will be evaluated with the implemented LES models for non-evaporating and evaporating diesel sprays. Different flavors of LES models such as dynamic structure and Smagorinsky will be compared against RANS approach and without any sub-grid scale (SGS) model. Task 2: This task will focus on extending the LES spray modeling approach for performing combustion simulations. Grid-convergence of LES combustion simulations will be assessed and recommendations will be provided minimum grid-sizes keeping in mind run-time vs. accuracy for LES modelers. Task 3: The above tasks will benchmark LES approach for full-cycle engine simulations. The recommended minimum cell sizes will be implemented for performing engine simulations. Cycle-to-cycle variations will be captured using the LES approach. Validation of the engine simulations will be performed against experimental data available in literature. Task 4: Since LES requires small resolutions the wall-clock times for the above tasks will be large. In general, such high wall-clock times are not affordable in the industry. The last task will be focus on further improving the SGS models to obtain flow structures even at coarser resolutions. Current: undetermined amount Justification: Requested: 150000 A specific reason has been given: We ran several large simulations of a 9-hole cummins injector which crashed unexpectedly. More core hours will help us finish couple of planned papers with Cummins as part of our DOE-CRADA project with them. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 27 Jan '15

27 Jan '15

Hello, A change in allocation has been requested: Requester: sinclair (Donald Sinclair) Project: Lattice-QCD Title: Lattice simulations of Conformal and Walking Technicolor. Description: We perform simulations to evaluate the functional integrals of QCD-like theories formulated on a discrete space-time lattice, to enable determination of the non-perturbative aspects these theories. These include the properties of these theories at non-zero temperature, including the scales of confinement and chiral symmetry breaking, and such zero temperature properties as spectra (including the Higgs mass), decay constants and the running of the gauge coupling constant. We are particularly interested in those theories where the coupling constant evolves very slowly, since these are candidate 'Walking Technicolor' theories. Related to these are theories with an infrared fixed point (conformal field theories). Our first goal is to differentiate between these two different types of behaviour for candidate theories. We are performing simulations of QCD-like theories which are models for Walking or Conformal Technicolor. We have been studying theories which are essentially QCD but with colour-sextet rather than colour-triplet quarks. We hope to measure the running of the QCD coupling constant. For 2 or 3 flavours, 2-loop perturbation theory predicts an infrared fixed point. For 2 flavours, it is possible that a chiral condensate forms before this fixed point is reached. If so, the fixed point is avoided, the theory is confining, and chiral symmetry breaks spontaneously. However, there is a region where the coupling constant evolves very slowly. These are the properties required for a walking technicolor theory. Simulations we have performed so far at finite temperature suggest that this theory might walk. So far the results are inconclusive. For 3 flavours we know that the theory should be conformal. Simulations at N_t=6 (12^3 X 6 lattice) and N_t=8 (16^3 X 8 lattice), some of which used Fusion, did not yet show evidence of conformality. We are thus moving to N_t=12 (24^3 X 12 lattice). We started these simulations on Blues, and intend to continue them in FY2015 on Blues, while performing lower mass simulations on Edison at NERSC. Our sextet quark codes are based on our earlier triplet quark codes and use the RHMC simulation method. The Rational Hybrid Monte Carlo (RHMC) is a stochastic molecular dynamics algorithm. The functional integral of QCD is written as a partition function of a classical field theory evolving in a fictitious time. The determinant of the Dirac operator raised to a fractional power is calculated by introducing bosonic fields (pseudofermions), and sandwiching this Dirac operator raised to minus said fractional power between them. This fractional power of the Dirac operator is approximated to machine accuracy by a rational approximation. After defining this this theory on a discrete space-time lattice, the inversions required by the partial-fraction expansion of the rational approximation are performed using Krylov space methods, in particular a multi-shift extension of the conjugate gradient algorithm. A global Metropolis Monte-Carlo accept/reject step applied at the end of each trajectory removes discretization errors introduced by the numerical integration of these stochastic equations of motion. We parallelize the code by assigning a fixed number of adjacent lattice sites to each MPI task. Network bandwidth ultimately limits how small a chunk of the lattice can be assigned to each task. Our 24^3 X 12 runs will be performed on 288 cores (18 nodes) of Blues. A short benchmark run of the 24^3 X 12 code yielded 667 Gflops or 2.3 Gflops/core. Since we have seen performances of > 3 Gflops/core on machines with similar processors, we suspect that at least 3 Gflops/core should be achievable on this machine. Since we have exhausted our allocation on Blues, we cannot perform more extensive tests. Scaling tests on Fusion and on Edison at NERSC for numbers of cores from 24 up to 288 cores indicate that the per core performance increases up to 144 or 288 cores, which we believe to be related to cache usage. Our new project, simulating QCD at finite mu using Complex Langevin with Gauge Cooling, will require some code development. Our old codes do not as yet have gauge cooling and our parallel code uses SHMEM rather than MPI for communication. Since the real Langevin is the limit of our Hybrid Molecular Dynamics code (the forerunner of the RHMC method) in which there is a single update per trajectory, most of what we have discussed for the RHMC applies, and will not be repeated here. We plan to use Blues and Carver at NERSC for the early, small-lattice simulations, 8^3 X 4 on 32 cores and 8^4 on 64 cores. Because of the smaller memory requirements, allowing more efficient cache usage, we expect to get a minimum performance of 3-4 Gflops/core for this code. Our requested allocation is based on running 1 288 core job half of the time at times when the NERSC machines are experiencing slowdowns for amall jobs. Current: undetermined amount Justification: We have not yet performed a detailed scaling analysis on Blues, however, we have such analyses performed on Fusion as well as Edison at NERSC. For Fusion, for our QCD with sextet quarks code on a 24^3 X 12 lattice running on Fusion we observed the following performances: 24 cores = 45 Gflops = 1.9 Gflops/core 48 cores = 95 Gflops = 2.0 Gflops/core 72 cores = 148 Gflops = 2.1 Gflops/core 96 cores = 220 Gflops = 2.3 Gflops/core 144 cores = 369 Gflops = 2.6 Gflops/core 288 cores = 784 Gflops = 2.7 Gflops/core For the same code and lattice size running on Edison at NERSC we observed the following performances. 24 cores = 53 Gflops = 2.2 Gflops/core 48 cores = 115 Gflops = 2.4 Gflops/core 72 cores = 191 Gflops = 2.7 Gflops/core 96 cores = 245 Gflops = 2.6 Gflops/core 144 cores = 510 Gflops = 3.5 Gflops/core 288 cores = 959 Gflops = 3.3 Gflops/core We use a custom assignment of tasks to nodes in order to minimize communications. Earlier recoding reduced the number of global reductions in the routines, which use most of the CPU time, by a factor of 2. The node codes are transparently vectorizable, to allow use of SSE instructions. Requested: 250000 A specific reason has been given: Estimated number of core-hours to complete current series of runs. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 27 Jan '15

27 Jan '15

Hello, A change in allocation has been requested: Requester: sinclair (Donald Sinclair) Project: Lattice-QCD Title: Lattice simulations of Conformal and Walking Technicolor. Description: We perform simulations to evaluate the functional integrals of QCD-like theories formulated on a discrete space-time lattice, to enable determination of the non-perturbative aspects these theories. These include the properties of these theories at non-zero temperature, including the scales of confinement and chiral symmetry breaking, and such zero temperature properties as spectra (including the Higgs mass), decay constants and the running of the gauge coupling constant. We are particularly interested in those theories where the coupling constant evolves very slowly, since these are candidate 'Walking Technicolor' theories. Related to these are theories with an infrared fixed point (conformal field theories). Our first goal is to differentiate between these two different types of behaviour for candidate theories. We are performing simulations of QCD-like theories which are models for Walking or Conformal Technicolor. We have been studying theories which are essentially QCD but with colour-sextet rather than colour-triplet quarks. We hope to measure the running of the QCD coupling constant. For 2 or 3 flavours, 2-loop perturbation theory predicts an infrared fixed point. For 2 flavours, it is possible that a chiral condensate forms before this fixed point is reached. If so, the fixed point is avoided, the theory is confining, and chiral symmetry breaks spontaneously. However, there is a region where the coupling constant evolves very slowly. These are the properties required for a walking technicolor theory. Simulations we have performed so far at finite temperature suggest that this theory might walk. So far the results are inconclusive. For 3 flavours we know that the theory should be conformal. Simulations at N_t=6 (12^3 X 6 lattice) and N_t=8 (16^3 X 8 lattice), some of which used Fusion, did not yet show evidence of conformality. We are thus moving to N_t=12 (24^3 X 12 lattice). We started these simulations on Blues, and intend to continue them in FY2015 on Blues, while performing lower mass simulations on Edison at NERSC. Our sextet quark codes are based on our earlier triplet quark codes and use the RHMC simulation method. The Rational Hybrid Monte Carlo (RHMC) is a stochastic molecular dynamics algorithm. The functional integral of QCD is written as a partition function of a classical field theory evolving in a fictitious time. The determinant of the Dirac operator raised to a fractional power is calculated by introducing bosonic fields (pseudofermions), and sandwiching this Dirac operator raised to minus said fractional power between them. This fractional power of the Dirac operator is approximated to machine accuracy by a rational approximation. After defining this this theory on a discrete space-time lattice, the inversions required by the partial-fraction expansion of the rational approximation are performed using Krylov space methods, in particular a multi-shift extension of the conjugate gradient algorithm. A global Metropolis Monte-Carlo accept/reject step applied at the end of each trajectory removes discretization errors introduced by the numerical integration of these stochastic equations of motion. We parallelize the code by assigning a fixed number of adjacent lattice sites to each MPI task. Network bandwidth ultimately limits how small a chunk of the lattice can be assigned to each task. Our 24^3 X 12 runs will be performed on 288 cores (18 nodes) of Blues. A short benchmark run of the 24^3 X 12 code yielded 667 Gflops or 2.3 Gflops/core. Since we have seen performances of > 3 Gflops/core on machines with similar processors, we suspect that at least 3 Gflops/core should be achievable on this machine. Since we have exhausted our allocation on Blues, we cannot perform more extensive tests. Scaling tests on Fusion and on Edison at NERSC for numbers of cores from 24 up to 288 cores indicate that the per core performance increases up to 144 or 288 cores, which we believe to be related to cache usage. Our new project, simulating QCD at finite mu using Complex Langevin with Gauge Cooling, will require some code development. Our old codes do not as yet have gauge cooling and our parallel code uses SHMEM rather than MPI for communication. Since the real Langevin is the limit of our Hybrid Molecular Dynamics code (the forerunner of the RHMC method) in which there is a single update per trajectory, most of what we have discussed for the RHMC applies, and will not be repeated here. We plan to use Blues and Carver at NERSC for the early, small-lattice simulations, 8^3 X 4 on 32 cores and 8^4 on 64 cores. Because of the smaller memory requirements, allowing more efficient cache usage, we expect to get a minimum performance of 3-4 Gflops/core for this code. Our requested allocation is based on running 1 288 core job half of the time at times when the NERSC machines are experiencing slowdowns for amall jobs. Current: undetermined amount Justification: We have not yet performed a detailed scaling analysis on Blues, however, we have such analyses performed on Fusion as well as Edison at NERSC. For Fusion, for our QCD with sextet quarks code on a 24^3 X 12 lattice running on Fusion we observed the following performances: 24 cores = 45 Gflops = 1.9 Gflops/core 48 cores = 95 Gflops = 2.0 Gflops/core 72 cores = 148 Gflops = 2.1 Gflops/core 96 cores = 220 Gflops = 2.3 Gflops/core 144 cores = 369 Gflops = 2.6 Gflops/core 288 cores = 784 Gflops = 2.7 Gflops/core For the same code and lattice size running on Edison at NERSC we observed the following performances. 24 cores = 53 Gflops = 2.2 Gflops/core 48 cores = 115 Gflops = 2.4 Gflops/core 72 cores = 191 Gflops = 2.7 Gflops/core 96 cores = 245 Gflops = 2.6 Gflops/core 144 cores = 510 Gflops = 3.5 Gflops/core 288 cores = 959 Gflops = 3.3 Gflops/core We use a custom assignment of tasks to nodes in order to minimize communications. Earlier recoding reduced the number of global reductions in the routines, which use most of the CPU time, by a factor of 2. The node codes are transparently vectorizable, to allow use of SSE instructions. Requested: 250000 A specific reason has been given: Estimated number of core-hours to complete current series of runs. This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0

[LCRC Accounts] Project Allocation Request
by accounts＠lcrc.anl.gov 26 Jan '15

26 Jan '15

Hello, A change in allocation has been requested: Requester: dduke (Daniel Duke) Project: XRayFuelSpray Title: Fuel Injection and Sprays Studied Using X-Ray Diagnostics Description: The cavitation modeling component of the project is being performed using the OpenFOAM framework. A number of state-of-the-art numerical solvers have been implemented in OpenFOAM; namely incompressible-liquid and fully compressible homogeneous relaxation cavitation models and more recently a fully compressible homogeneous relaxation cavitation model which includes non-condensable gas modeling (HRMFoam). These codes were developed by Schmidt et al at the University of Massachusetts-Amherst, and we continue to collaborate with them on code development and efficiency improvements. In the last year, we have made some substantial improvements to the cavitation model and the parallel efficiency of its implementation on Blues/Fusion. With the support of LCRC technical staff and our collaborators at the University of Massachusetts-Amherst, we have implemented a custom build of OpenFOAM-3.0 on Blues against the system native MPI, and we have made the necessary changes to the HRMFoam solver’s source code in order to move from the older OpenFOAM-1.6 to the new 3.0 framework. This has led to substantial gains in parallel performance; up to 20% on 32 cores and 12% on 64 cores on Blues. A scalability study has been performed with the new code, and details are given in the Large Allocation Efficiency statement. In the new OpenFOAM-3.0 build, we have successfully implemented both Large Eddy Simulation (LES) and Realizable k-epsilon (KE) turbulence models. Furthermore, we have implemented code which allows us to take virtual projections of the three-dimensional simulations so that they may be quantitatively compared with the x-ray experiments. In the following year, we plan to extend high-fidelity cavitation simulations by undertaking longer simulation times in order to capture lower-frequency hydrodynamic phenomena which we have observed in our experiments. Improvements in efficiency will allows us to pursue longer run-time simulations. We also intend to simulate the effects of varying pressure and flow rate on cavitation, in order to compare against new x-ray fluorescence data from APS. The gas jet simulations which we plan to undertake in Q1 are performed with the CONVERGE CFD software, its use is already well established at LCRC. Tomographic reconstruction of gasoline direct injection spray measurements from APS will be conducted using the TomoPy software package developed at APS. We intend to run single-node jobs to efficiently perform multiple reconstructions of large (multi-GB) datasets. We have also successfully compiled a Paraview server on our Fusion dedicated nodes which allows us to rapidly visualize the results of our simulations and reconstructions. Using Blues and Fusion will allow us to achieve rapid turnaround of results during beamtime. This will be a significant aid to the experimental program. As the quantity of tomography work increases, we may consider purchasing more dedicated nodes. Current: undetermined amount Justification: Our requested allocation for next year will exceed 0.5M core-hours in order to encompass the gas jet simulations in Q1 and tomographic reconstruction work in Q2-4. The majority of core hours will still be dedicated to OpenFOAM simulations, which will be undertaken on both dedicated Fusion nodes and also on Blues, using OpenFOAM-3.0. We have demonstrated notable improvements in scalability on Blues as detailed above. A scalability study of OpenFOAM-3.0 performance with multi-million cell meshes has been undertaken, demonstrating excellent linear scalability. We observe 81% efficiency relative to single-CPU performance, scaling linearly up to 320 cores (20 nodes) with a 15-million cell mesh (a typical upper limit). We have implemented Scotch decomposition in order to ensure good load balancing & minimization of cross-processor communication boundaries. The present OpenFOAM simulations use static meshes, so load rebalancing at runtime is not a concern. We intend to run typical job sizes of 64 to 128 cores; smaller than the maximum job size for which good scalability has been demonstrated. Further details regarding scalability studies can be obtained by contacting the project PIs. Requested: 50000 A specific reason has been given: We have had to accelerate our schedule for completing OpenFOAM CFD calculations this year in order to meet a number of very early conference paper deadlines. As such, we have already exhausted our allocation rather early. We would like to request an addition 50k hours this quarter, to complete the final simulations in order to meet our publication deadline. If you have any questions or require further details please contact PI Duke (dduke(a)anl.gov) This needs to be approved and the final allocation amount decided upon. Thank You, The LCRC Accounts System

1 0