October 2022 - Ecp-aurora-sdk-discuss

Re: [Ecp-aurora-sdk-discuss] Update to default oneapi module file
by Muralidharan, Servesh 20 Oct '22

20 Oct '22

There are two important changes that are planned along with the default oneapi SDK module update today. Firstly, When this default modulefile change takes place, the nodes will also move to a new image with an updated KMD among other changes. After this image change its preferred that users migrate to one of the new module files listed below for optimal performance based on the recommendation below, Current New Recommendation PVC AOT String (For all Florentia nodes in JLSE) oneapi/eng-compiler/2022.06.30.002 (default) oneapi/eng-compiler/2022.06.30.006 (default) 12.60.7 oneapi/eng-compiler/2022.06.30.004 oneapi/eng-compiler/2022.06.30.006 12.60.7 oneapi/eng-compiler/2022.10.15.002 (newest SDK) oneapi/eng-compiler/2022.10.15.004 (newest SDK) 12.60.7 oneapi/release/2022.06.30.001 oneapi/release/2022.06.30.005 12.60.7 oneapi/release/2022.06.30.003 oneapi/release/2022.06.30.005 12.60.7 oneapi/release/2022.10.15.001 oneapi/release/2022.10.15.003 12.60.7 Should you have specific issues with your app that arises due to this change, please do reach out. Priority bugs and known issues are tracked as before. Secondly, We will also be adding new nodes to the queue when this migration happens. There are still pending updates on the nodes so expect us to be monitoring them and taking them out of queue to complete these, as necessary. There will be two different SKUs of the PVC available to users on the Florentia systems at JLSE organized by the queues. Please take a look at the wiki here for details: https://wiki.jlse.anl.gov/display/inteldga/Florentia On behalf of the early hw team. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Thursday, October 20, 2022 at 12:04 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Update to default oneapi module file New modulefiles are available for use on JLSE. These contain an updated VTune, MPICH 49.1 and newer Level Zero driver. The default oneAPI modulefile will be changed to use oneapi/eng-compiler/2022.06.30.006 at 5pm CDT today. The module files for this release will be: oneapi/release/2022.10.15.003 oneapi/eng-compiler/2022.06.30.006 oneapi/release/2022.10.15.003 oneapi/eng-compiler/2022.10.15.004 -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Update to default oneapi module file
by Chan-nui, Christopher 20 Oct '22

20 Oct '22

New modulefiles are available for use on JLSE. These contain an updated VTune, MPICH 49.1 and newer Level Zero driver. The default oneAPI modulefile will be changed to use oneapi/eng-compiler/2022.06.30.006 at 5pm CDT today. The module files for this release will be: oneapi/release/2022.10.15.003 oneapi/eng-compiler/2022.06.30.006 oneapi/release/2022.10.15.003 oneapi/eng-compiler/2022.10.15.004 -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] E4S Usage in JLSE on Intel GPUs
by Knight, Christopher J. 19 Oct '22

19 Oct '22

Thanks to those that replied. Very helpful! Feedback from others is helpful with this initial planning. chris On Oct 14, 2022, at 12:37 PM, Knight, Christopher J. <knightc(a)anl.gov<mailto:[email protected]>> wrote: Hi everyone, Discussions and preparations are ramping up to get the E4S module in good shape for PVC in Sunspot/Aurora and some feedback from teams working on the testbed would be useful to have in this process. Some basic info on how you are using (or not) the E4S module is helpful here. 1. If your application/workflow uses a Software Technology (ST) product packaged as part of E4S, then... *) What E4S ST products does your work depend on? *) Are you using the E4S module in JLSE (likely in work on Arcticus)? *) If you are not using the E4S module, then why not? For example, “Using kokkos modules provided by Tim built against latest SDK”. *) Are there other variants or options for the packages required for your work not currently provided in E4S? 2. Some feedback on which E4S Products should initially be supported on Sunspot/Aurora would be helpful. Please let us know if something critical is missing from the following *draft* list. Other ST products will certainly be supported, but we want to identify a prioritized list to ensure teams are not initially blocked waiting on something. Data & Viz: ADIOS, Darshan, HDF5, PnetCDF, UnifyFS, VTK-m Math Libraries: BLAS/MAGMA, Ginkgo, hypre, kokkos-kernels, PETSc/TAO, ScaLAPACK/SLATE, STRUMPACK, SUNDIALS, SuperLU, Trilinos, FFT/heFFTe PMR: Kokkos, Legion, RAJA, Umpire, MPICH Tools: HPCToolkit, PAPI, TAU Any feedback here is appreciated. Feel free to reply directly to me. chris ============ $ module load e4s $ module avail ----------------------------------------- /soft/ecp/ParaTools/E4S/22.08/spack/share/spack/modules/linux-opensuse_leap15-icelake ------------------------------------------ adios/1.13.1-oneapi-2022.1.0-mpi gmp/6.2.1-oneapi-2022.1.0 papyrus/1.0.2-oneapi-2022.1.0-mpi umap/2.1.0-oneapi-2022.1.0 alquimia/1.0.9-oneapi-2022.1.0-mpi gotcha/1.0.3-oneapi-2022.1.0 parallel-netcdf/1.12.2-oneapi-2022.1.0-mpi umpire/2022.03.1-oneapi-2022.1.0 aml/0.2.0-oneapi-2022.1.0 hdf5-vol-async/1.2-oneapi-2022.1.0-mpi parsec/3.0.2012-oneapi-2022.1.0-mpi upcxx/2022.3.0-oneapi-2022.1.0-mpi aml/0.2.0-oneapi-2022.1.0-level-zero hdf5/1.12.2-oneapi-2022.1.0-mpi petsc/3.17.4-oneapi-2022.1.0-mpi veloc/1.5-oneapi-2022.1.0-mpi amrex/22.08-oneapi-2022.1.0-mpi heffte/2.2.0-oneapi-2022.1.0-mpi plasma/21.8.29-oneapi-2022.1.0 vtk-m/1.8.0-oneapi-2022.1.0-mpi amrex/22.08-oneapi-2022.1.0-mpi-sycl hypre/2.25.0-oneapi-2022.1.0-mpi plumed/2.8.0-oneapi-2022.1.0-mpi wannier90/3.1.0-oneapi-2022.1.0-mpi arborx/1.2-oneapi-2022.1.0-mpi kokkos-kernels/3.6.00-oneapi-2022.1.0 precice/2.5.0-oneapi-2022.1.0-mpi zfp/0.5.5-oneapi-2022.1.0 archer/2.0.0-oneapi-2022.1.0 kokkos-kernels/3.6.00-oneapi-2022.1.0-sycl pumi/2.2.7-oneapi-2022.1.0-mpi argobots/1.1-oneapi-2022.1.0 kokkos/3.6.00-oneapi-2022.1.0-sycl py-jupyterhub/1.4.1-oneapi-2022.1.0 bolt/2.0-oneapi-2022.1.0 kokkos/3.6.01-oneapi-2022.1.0 qthreads/1.16-oneapi-2022.1.0 butterflypack/2.1.1-oneapi-2022.1.0-mpi lammps/20220623-oneapi-2022.1.0-mpi scr/3.0-oneapi-2022.1.0-mpi cabana/0.5.0-oneapi-2022.1.0-mpi legion/21.03.0-oneapi-2022.1.0 slepc/3.17.2-oneapi-2022.1.0-mpi cabana/0.5.0-oneapi-2022.1.0-mpi-sycl libnrm/0.1.0-oneapi-2022.1.0-mpi stc/0.9.0-oneapi-2022.1.0-mpi chai/2022.03.0-oneapi-2022.1.0 libquo/1.3.1-oneapi-2022.1.0-mpi strumpack/6.3.1-oneapi-2022.1.0-mpi charliecloud/0.27-oneapi-2022.1.0 libunwind/1.6.2-oneapi-2022.1.0 sundials/6.2.0-oneapi-2022.1.0-mpi conduit/0.8.3-oneapi-2022.1.0-mpi loki/0.1.7-oneapi-2022.1.0 superlu-dist/7.2.0-oneapi-2022.1.0-mpi darshan-runtime/3.4.0-oneapi-2022.1.0-mpi mercury/2.1.0-oneapi-2022.1.0-mpi superlu/5.3.0-oneapi-2022.1.0 darshan-util/3.4.0-oneapi-2022.1.0 metall/0.21-oneapi-2022.1.0-mpi swig/4.0.2-fortran-oneapi-2022.1.0 datatransferkit/3.1-rc3-oneapi-2022.1.0-mpi mfem/4.4.0-oneapi-2022.1.0-mpi swig/4.0.2-oneapi-2022.1.0 faodel/1.2108.1-oneapi-2022.1.0-mpi mpark-variant/1.4.0-oneapi-2022.1.0 sz/2.1.12.2-oneapi-2022.1.0 flit/2.1.0-oneapi-2022.1.0 mpifileutils/0.11.1-oneapi-2022.1.0-mpi tasmanian/7.7-oneapi-2022.1.0-mpi fortrilinos/2.0.0-oneapi-2022.1.0-mpi nccmp/1.9.0.1-oneapi-2022.1.0-mpi tau/2.31.1-oneapi-2022.1.0-mpi gasnet/2022.3.0-oneapi-2022.1.0 nco/5.0.1-oneapi-2022.1.0-mpi tau/2.31.1-oneapi-2022.1.0-mpi-level-zero ginkgo/1.4.0-oneapi-2022.1.0 netlib-scalapack/2.2.0-oneapi-2022.1.0-mpi trilinos/13.0.1-oneapi-2022.1.0-mpi globalarrays/5.8-oneapi-2022.1.0-mpi omega-h/9.34.1-oneapi-2022.1.0-mpi turbine/1.3.0-oneapi-2022.1.0-mpi …

1 0

E4S Usage in JLSE on Intel GPUs
by Knight, Christopher J. 14 Oct '22

14 Oct '22

Hi everyone, Discussions and preparations are ramping up to get the E4S module in good shape for PVC in Sunspot/Aurora and some feedback from teams working on the testbed would be useful to have in this process. Some basic info on how you are using (or not) the E4S module is helpful here. 1. If your application/workflow uses a Software Technology (ST) product packaged as part of E4S, then... *) What E4S ST products does your work depend on? *) Are you using the E4S module in JLSE (likely in work on Arcticus)? *) If you are not using the E4S module, then why not? For example, “Using kokkos modules provided by Tim built against latest SDK”. *) Are there other variants or options for the packages required for your work not currently provided in E4S? 2. Some feedback on which E4S Products should initially be supported on Sunspot/Aurora would be helpful. Please let us know if something critical is missing from the following *draft* list. Other ST products will certainly be supported, but we want to identify a prioritized list to ensure teams are not initially blocked waiting on something. Data & Viz: ADIOS, Darshan, HDF5, PnetCDF, UnifyFS, VTK-m Math Libraries: BLAS/MAGMA, Ginkgo, hypre, kokkos-kernels, PETSc/TAO, ScaLAPACK/SLATE, STRUMPACK, SUNDIALS, SuperLU, Trilinos, FFT/heFFTe PMR: Kokkos, Legion, RAJA, Umpire, MPICH Tools: HPCToolkit, PAPI, TAU Any feedback here is appreciated. Feel free to reply directly to me. chris ============ $ module load e4s $ module avail ----------------------------------------- /soft/ecp/ParaTools/E4S/22.08/spack/share/spack/modules/linux-opensuse_leap15-icelake ------------------------------------------ adios/1.13.1-oneapi-2022.1.0-mpi gmp/6.2.1-oneapi-2022.1.0 papyrus/1.0.2-oneapi-2022.1.0-mpi umap/2.1.0-oneapi-2022.1.0 alquimia/1.0.9-oneapi-2022.1.0-mpi gotcha/1.0.3-oneapi-2022.1.0 parallel-netcdf/1.12.2-oneapi-2022.1.0-mpi umpire/2022.03.1-oneapi-2022.1.0 aml/0.2.0-oneapi-2022.1.0 hdf5-vol-async/1.2-oneapi-2022.1.0-mpi parsec/3.0.2012-oneapi-2022.1.0-mpi upcxx/2022.3.0-oneapi-2022.1.0-mpi aml/0.2.0-oneapi-2022.1.0-level-zero hdf5/1.12.2-oneapi-2022.1.0-mpi petsc/3.17.4-oneapi-2022.1.0-mpi veloc/1.5-oneapi-2022.1.0-mpi amrex/22.08-oneapi-2022.1.0-mpi heffte/2.2.0-oneapi-2022.1.0-mpi plasma/21.8.29-oneapi-2022.1.0 vtk-m/1.8.0-oneapi-2022.1.0-mpi amrex/22.08-oneapi-2022.1.0-mpi-sycl hypre/2.25.0-oneapi-2022.1.0-mpi plumed/2.8.0-oneapi-2022.1.0-mpi wannier90/3.1.0-oneapi-2022.1.0-mpi arborx/1.2-oneapi-2022.1.0-mpi kokkos-kernels/3.6.00-oneapi-2022.1.0 precice/2.5.0-oneapi-2022.1.0-mpi zfp/0.5.5-oneapi-2022.1.0 archer/2.0.0-oneapi-2022.1.0 kokkos-kernels/3.6.00-oneapi-2022.1.0-sycl pumi/2.2.7-oneapi-2022.1.0-mpi argobots/1.1-oneapi-2022.1.0 kokkos/3.6.00-oneapi-2022.1.0-sycl py-jupyterhub/1.4.1-oneapi-2022.1.0 bolt/2.0-oneapi-2022.1.0 kokkos/3.6.01-oneapi-2022.1.0 qthreads/1.16-oneapi-2022.1.0 butterflypack/2.1.1-oneapi-2022.1.0-mpi lammps/20220623-oneapi-2022.1.0-mpi scr/3.0-oneapi-2022.1.0-mpi cabana/0.5.0-oneapi-2022.1.0-mpi legion/21.03.0-oneapi-2022.1.0 slepc/3.17.2-oneapi-2022.1.0-mpi cabana/0.5.0-oneapi-2022.1.0-mpi-sycl libnrm/0.1.0-oneapi-2022.1.0-mpi stc/0.9.0-oneapi-2022.1.0-mpi chai/2022.03.0-oneapi-2022.1.0 libquo/1.3.1-oneapi-2022.1.0-mpi strumpack/6.3.1-oneapi-2022.1.0-mpi charliecloud/0.27-oneapi-2022.1.0 libunwind/1.6.2-oneapi-2022.1.0 sundials/6.2.0-oneapi-2022.1.0-mpi conduit/0.8.3-oneapi-2022.1.0-mpi loki/0.1.7-oneapi-2022.1.0 superlu-dist/7.2.0-oneapi-2022.1.0-mpi darshan-runtime/3.4.0-oneapi-2022.1.0-mpi mercury/2.1.0-oneapi-2022.1.0-mpi superlu/5.3.0-oneapi-2022.1.0 darshan-util/3.4.0-oneapi-2022.1.0 metall/0.21-oneapi-2022.1.0-mpi swig/4.0.2-fortran-oneapi-2022.1.0 datatransferkit/3.1-rc3-oneapi-2022.1.0-mpi mfem/4.4.0-oneapi-2022.1.0-mpi swig/4.0.2-oneapi-2022.1.0 faodel/1.2108.1-oneapi-2022.1.0-mpi mpark-variant/1.4.0-oneapi-2022.1.0 sz/2.1.12.2-oneapi-2022.1.0 flit/2.1.0-oneapi-2022.1.0 mpifileutils/0.11.1-oneapi-2022.1.0-mpi tasmanian/7.7-oneapi-2022.1.0-mpi fortrilinos/2.0.0-oneapi-2022.1.0-mpi nccmp/1.9.0.1-oneapi-2022.1.0-mpi tau/2.31.1-oneapi-2022.1.0-mpi gasnet/2022.3.0-oneapi-2022.1.0 nco/5.0.1-oneapi-2022.1.0-mpi tau/2.31.1-oneapi-2022.1.0-mpi-level-zero ginkgo/1.4.0-oneapi-2022.1.0 netlib-scalapack/2.2.0-oneapi-2022.1.0-mpi trilinos/13.0.1-oneapi-2022.1.0-mpi globalarrays/5.8-oneapi-2022.1.0-mpi omega-h/9.34.1-oneapi-2022.1.0-mpi turbine/1.3.0-oneapi-2022.1.0-mpi …

1 0

Re: [Ecp-aurora-sdk-discuss] OpenCL on recent oneAPI releases
by Elliott Slaughter 06 Oct '22

06 Oct '22

Please disregard my last message, I was trying to run a version of the code that assumes the kernels are pre-compiled to SPIR-V (without precompiling to SPIR-V). The version that loads OpenCL code from a string runs fine. On Thu, Oct 6, 2022 at 3:46 PM Elliott Slaughter <slaughter(a)cs.stanford.edu> wrote: > Hi, > > I have a set of GPU portability samples here [1] that I maintain. They're > written in several languages, including OpenCL. > > [1]: https://github.com/ExaFEL/gpu-portability-samples > > For the life of me, I cannot get the OpenCL version to work on recent > oneAPI. It's been a while since I tested; I guess the opencl module is gone > now. I tried loading oneapi instead but I get crashes every way I try to > run the code. > > In theory, something like the following ought to do it: > > git clone https://github.com/ExaFEL/gpu-portability-samples.git > qsub -I -n 1 -t 60 -q arcticus > cd gpu-portability-samples/opencl > source arcticus_env.sh > make CC=$CC CXX=$CXX > > (This loads the oneapi module and uses the default CC/CXX compilers. The > exact compiler shouldn't really matter because it loads the OpenCL kernel > at runtime anyway.) > > When I try to run on CPU: > > $ ./saxpy -d 0 > Running on platform: Intel(R) OpenCL > Running on device: Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz > CL_DEVICE_ADDRESS_BITS is 64 for this device. > error -30 > saxpy: saxpy_main.cc:132: int main(int, char **): Assertion `false' failed. > Aborted > > When I instead try to run on GPU: > > $ ./saxpy -d 1 > Running on platform: Intel(R) OpenCL > Segmentation fault > > This is a code that worked some time ago, not just on Intel's software > stack but on AMD as well. > > I hope I'm just missing something obvious, but I don't see it right now. > > Thanks in advance! > > -- > Elliott Slaughter > > "Don't worry about what anybody else is going to do. The best way to > predict the future is to invent it." - Alan Kay > -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay

1 0

OpenCL on recent oneAPI releases
by Elliott Slaughter 06 Oct '22

06 Oct '22

Hi, I have a set of GPU portability samples here [1] that I maintain. They're written in several languages, including OpenCL. [1]: https://github.com/ExaFEL/gpu-portability-samples For the life of me, I cannot get the OpenCL version to work on recent oneAPI. It's been a while since I tested; I guess the opencl module is gone now. I tried loading oneapi instead but I get crashes every way I try to run the code. In theory, something like the following ought to do it: git clone https://github.com/ExaFEL/gpu-portability-samples.git qsub -I -n 1 -t 60 -q arcticus cd gpu-portability-samples/opencl source arcticus_env.sh make CC=$CC CXX=$CXX (This loads the oneapi module and uses the default CC/CXX compilers. The exact compiler shouldn't really matter because it loads the OpenCL kernel at runtime anyway.) When I try to run on CPU: $ ./saxpy -d 0 Running on platform: Intel(R) OpenCL Running on device: Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz CL_DEVICE_ADDRESS_BITS is 64 for this device. error -30 saxpy: saxpy_main.cc:132: int main(int, char **): Assertion `false' failed. Aborted When I instead try to run on GPU: $ ./saxpy -d 1 Running on platform: Intel(R) OpenCL Segmentation fault This is a code that worked some time ago, not just on Intel's software stack but on AMD as well. I hope I'm just missing something obvious, but I don't see it right now. Thanks in advance! -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay

1 0