November 2022 - Ecp-aurora-sdk-discuss

DL Frameworks 2022.4 release now available in jlse
by Ramer, Rebecca 23 Nov '22

23 Nov '22

Hi, We would like to announce availability of new frameworks releases for use on the florentia systems in the JLSE. This release is based on the 2022.4.0 versions of the DeepLearning Frameworks. These versions of pytorch and tensorflow can be run on on PVC XT systems only, i.e. florentia nodes. The new releases are available via: frameworks/2022.11.15.001 which supports implicit scaling frameworks/2022.11.15.002 which supports tile as device Additional information on setting up and working with this update can be found here for pytorch: https://wiki.jlse.anl.gov/display/inteldga/2022.4.0+-+Pytorch and here for tensorflow: https://wiki.jlse.anl.gov/display/inteldga/2022.4.0+-+Tensorflow Please let us know of any issues encountered. Happy Thanksgiving! Rebecca Ramer rebecca.ramer(a)intel.com<mailto:[email protected]>

1 0

New kokkos modules using 2022.10.15.006/eng-compiler and more recent kokkos source versions
by Williams, Timothy J. 18 Nov '22

18 Nov '22

There are new default kokkos modules built using oneapi/2022.10.15.006/eng-compiler, which you will get if you do module load kokkos/sycl_intel_aot module load kokkos/sycl_intel module load kokkos/openmptarget_intel_aot module load kokkos/openmptarget_intel (Just module load kokkos will default to module load kokkos/sycl_intel_aot). Important notes: 1. The AoT versions are built only for PVC (device 12.60.7). They no longer also include ATS. 2. The source code version for kokkos was updated to something substantially more recent. If either of these, especially the newer kokkos version, causes build or runtime errors for you, please let me know. Best regards, Tim ------- Timothy J. Williams, Ph.D. --- Argonne National Laboratory --- tjwilliams(a)anl.gov -------

1 0

Re: [Ecp-aurora-sdk-discuss] Updating default oneapi modules on JLSE/Sunspot
by Chan-nui, Christopher 16 Nov '22

16 Nov '22

The default modulefile has now been updated. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Wednesday, November 16, 2022 at 8:19 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Updating default oneapi modules on JLSE/Sunspot At noon Central time the default oneAPI module file will be updated to the latest engineering compiler version. Remember there are breaking changes with regards to SYCL2020 software. Software that sits above Compiler+MKL may be affected. The new module file is: oneapi/eng-compiler/2022.10.15.006 -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Updating default oneapi modules on JLSE/Sunspot
by Chan-nui, Christopher 15 Nov '22

15 Nov '22

At noon Central time the default oneAPI module file will be updated to the latest engineering compiler version. Remember there are breaking changes with regards to SYCL2020 software. Software that sits above Compiler+MKL may be affected. The new module file is: oneapi/eng-compiler/2022.10.15.006 -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] New engineering compiler and MKL available
by Bertoni, Colleen 07 Nov '22

07 Nov '22

Hi everyone, For bug reproducers, there were 22 fixes and 2 regressions (see the list below) on florentia, so this looks like a really nice drop! However, we did need to update many of the tests for the breaking SYCL2020 changes. Chris summarized them in his email, and we put a list of the issues we needed to fix in the reproducers below. For apps, there were seven application regressions, likely due to to the SYCL2020 changes that Chris mentioned. The people who contributed apps (thanks a lot!) should have received an email about the breaking changes. If you have a chance to update for these changes it would be appreciated! Thanks, Colleen Changes we made in the reproducers, which you should make in your applications if needed: ``` 0) "dpcpp" will be deprecated soon, please use "icpx -fsycl" 1) "cl::sycl" namespace is gone, just use "sycl::". For example: ``` - cl::sycl::buffer<float> d_x { &x, 1}; - cl::sycl::buffer<float> d_y { &y, 1}; - cl::sycl::buffer<float> d_a { &A, 1}; + sycl::buffer<float> d_x { &x, 1}; + sycl::buffer<float> d_y { &y, 1}; + sycl::buffer<float> d_a { &A, 1}; ``` 2) local accessors changed ("sycl::access::target::local -> sycl::local_accessor"). For example: ``` -using localAcc = cl::sycl::accessor<cl::sycl::cl_double, 2, cl::sycl::access::mode::read_write, - cl::sycl::access::target::local>; - +using localAcc = sycl::local_accessor<sycl::cl_double, 2>; ``` 3) Selectors changed (cpu/gpu_selector --> cpu/gpu_selector_v). For example: ``` - sycl::queue q(cl::sycl::gpu_selector()); + sycl::queue q(sycl::gpu_selector_v); ``` 4) `max_work_item_sizes` from dev.get_info now takes in a parameter for the dimensions. For example: ``` - << sel_dev.get_info<sycl::info::device::max_work_item_sizes>().get(0) - << " " << sel_dev.get_info<sycl::info::device::max_work_item_sizes>().get(1) - << " " << sel_dev.get_info<sycl::info::device::max_work_item_sizes>().get(2) + << sel_dev.get_info<sycl::info::device::max_work_item_sizes<3>>().get(0) + << " " << sel_dev.get_info<sycl::info::device::max_work_item_sizes<3>>().get(1) + << " " << sel_dev.get_info<sycl::info::device::max_work_item_sizes<3>>().get(2) ``` 5) Note that for OpenCL types, it should change like: ```cl::sycl::cl_float --> sycl::opencl::cl_float``` by the Spec, but right now it doesn't seem to work. however. ```cl::sycl::cl_float --> sycl::cl_float``` works. From Thomas: " I would suggest you not use cl_int and friends and rely more on the associated std datatype (`std::uint_32t` and the like). It will make your code more portable and more c++ isy :) " ``` Fixes with 2022.10.15.005 compared to 2022.10.004: ``` source/reproducers/dpcpp/ATSsubdevice_regression source/reproducers/dpcpp/NWChemEx-CCSDT source/reproducers/dpcpp/complex_math_functions [CMPLRLLVM-24169] source/reproducers/dpcpp/complex_math_functions:complex_math_functions_workarround1 [CMPLRLLVM-24169] source/reproducers/dpcpp/complex_math_functions:complex_math_functions_workarround2 [CMPLRLLVM-24169] source/reproducers/dpcpp/complex_math_functions:complex_math_functions_workarround3 [CMPLRLLVM-24169] source/reproducers/dpcpp/default_should_be_device:default_should_be_device_ath_run [CMPLRLLVM-40380] source/reproducers/dpcpp/oneDPL_incl_excl_scan_inplace [ONEDPL-490] P0 source/reproducers/ifx/select_type_all_sky [CMPLRLLVM-27838] source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 [CMPLRLLVM-39495] source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 [CMPLRLLVM-39495] source/reproducers/mkl/getrs_strided_group_compare:getrs_strided_group_compare_ath_run [MKLD-12830,XDEPS-4370] P0 source/reproducers/mkl/getrs_strided_group_compare:getrs_strided_group_compare_ze_binary_ath_run [MKLD-12830,XDEPS-4370] P0 source/reproducers/openmp/fmm_ice_array_reduction source/reproducers/openmp/fortran_array_reduction source/reproducers/openmp/fortran_asm_link:fortran_asm_link_elpa [CMPLRLLVM-39846] source/reproducers/openmp/kokkos.test_view:kokkos.test_view_is_device_ptr [CMPLRLLVM-23398,CMPLRLLVM-38777] P0 source/reproducers/openmp/mo_cloud_optics_ICE [CMPLRLLVM-26260,CMPLRLLVM-33311] source/reproducers/openmp/omp_interop_destroy:omp_interop_destroy_maptofrom [CMPLRLLVM-39114] source/reproducers/openmp/omp_simd_ice_qmc [CMPLRLLVM-39345] source/reproducers/openmp/short-vector-math:short-vector-math.r.O3.fastmath.fiopenmp [CMPLRLLVM-11184,CMPLRLLVM-18505,CMPLRLLVM-20040,CMPLRLLVM-21077] P0 source/reproducers/openmp/short-vector-math:short-vector-math.r.fiopenmp [CMPLRLLVM-11184,CMPLRLLVM-18505,CMPLRLLVM-20040,CMPLRLLVM-21077] P0 ``` Regressions with 2022.10.15.005 compared to 2022.10.004: ``` source/reproducers/dpcpp/float_calc (precision issue) source/reproducers/dpcpp/test_xgc_get_acoef (CMPLRLLVM-40905) ``` ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Friday, November 4, 2022 5:55 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] New engineering compiler and MKL available Hello, We are pleased to release a new version of the engineering compilers and MKL. This has NOT been made default as there are breaking changes with regards to SYCL2020 software. Expect software that sits above the Compiler+MKL to break. It is available as oneapi/eng-compiler/2022.10.15.005 RELEASE NOTES: Compiler Release Notes: EXTREMELY IMPORTANT: Remove cl:: namespace from sycl/sycl.hpp header Promote SYCL2020_CONFORMANT_APIS guarded changes for ABI-breaking release. SYCL2020 conformance is enabled by default, expect breaking changes to your code. All current features hidden behind SYCL2020_CONFORM_APIS are now enabled by default. Future breaking changes will continue to be guarded by this flag. If your code is SYCL2020 conformant we recommend leaving this flag enabled. sycl 2020: Implement SYCL 2020 info descriptors as types Implement sycl::buffer_allocator which takes a template argument Removals sycl 1.2.1: Remove deprecated sycl::program class Remove SYCL1.2.1 reductions Remove old implementation of half type Memory manager removals sycl Handler ABI removals remove make_contex and make_queue versions Remove deprecated make_queue input type for L0 Remove get_native methods from sycl classes Remove kernel::get_work_group_info Remove host device from DPC++ Refactors: Refactor accessor to make it more ABI stable Refactor buffer and image to make them more ABI stable CG.hpp refactors and changes Miscellaneous: Revisit exported symbols platform() return a platform with no device Add no_except to common has_property member functions Has_context() is now no_except Add missing SYCL common property interface to accessors fsycl-dead-args-optimization fails to eliminate one dead argument per accessor MKL Release Notes LAPACK Improved performance of getrs_batch group APIs on GPU FFT Support for customer-allocated scratch space added Improved performance for small 1D real-to-complex and complex-to-real problems, targeting especially sizes relevant to the GROMACS workload Improved PVC performance for 1D complex FFTs for targeted small even sizes related to West and Quantum ESPRESSO workloads Enabled real-to-complex and complex-to-real support through OpenMP offload with the C FFTW interface. (This change introduced a bug that breaks single dynamic library linking with FFTW interfaces that was fixed for 2023.0 after Oct. 17th) Sparse Incremental update to sparse::matmat() performance for PETSc/Kokkos workloads Incremental improvement to sparse::gemm() for ThunderSVM workload -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

New engineering compiler and MKL available
by Chan-nui, Christopher 04 Nov '22

04 Nov '22

Hello, We are pleased to release a new version of the engineering compilers and MKL. This has NOT been made default as there are breaking changes with regards to SYCL2020 software. Expect software that sits above the Compiler+MKL to break. It is available as oneapi/eng-compiler/2022.10.15.005 RELEASE NOTES: Compiler Release Notes: EXTREMELY IMPORTANT: Remove cl:: namespace from sycl/sycl.hpp header Promote SYCL2020_CONFORMANT_APIS guarded changes for ABI-breaking release. SYCL2020 conformance is enabled by default, expect breaking changes to your code. All current features hidden behind SYCL2020_CONFORM_APIS are now enabled by default. Future breaking changes will continue to be guarded by this flag. If your code is SYCL2020 conformant we recommend leaving this flag enabled. sycl 2020: Implement SYCL 2020 info descriptors as types Implement sycl::buffer_allocator which takes a template argument Removals sycl 1.2.1: Remove deprecated sycl::program class Remove SYCL1.2.1 reductions Remove old implementation of half type Memory manager removals sycl Handler ABI removals remove make_contex and make_queue versions Remove deprecated make_queue input type for L0 Remove get_native methods from sycl classes Remove kernel::get_work_group_info Remove host device from DPC++ Refactors: Refactor accessor to make it more ABI stable Refactor buffer and image to make them more ABI stable CG.hpp refactors and changes Miscellaneous: Revisit exported symbols platform() return a platform with no device Add no_except to common has_property member functions Has_context() is now no_except Add missing SYCL common property interface to accessors fsycl-dead-args-optimization fails to eliminate one dead argument per accessor MKL Release Notes LAPACK Improved performance of getrs_batch group APIs on GPU FFT Support for customer-allocated scratch space added Improved performance for small 1D real-to-complex and complex-to-real problems, targeting especially sizes relevant to the GROMACS workload Improved PVC performance for 1D complex FFTs for targeted small even sizes related to West and Quantum ESPRESSO workloads Enabled real-to-complex and complex-to-real support through OpenMP offload with the C FFTW interface. (This change introduced a bug that breaks single dynamic library linking with FFTW interfaces that was fixed for 2023.0 after Oct. 17th) Sparse Incremental update to sparse::matmat() performance for PETSc/Kokkos workloads Incremental improvement to sparse::gemm() for ThunderSVM workload -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

DL Frameworks 2022.3 release now supports PVC systems
by Ramer, Rebecca 02 Nov '22

02 Nov '22

Hi, We would like to announce availability of additional frameworks modules for pytorch and tensorflow for use on the florentia systems in the JLSE. This release is based on the 2022.3.0 versions of the DeepLearning Frameworks. These versions of the frameworks can be run on on PVC XT systems only, i.e. florentia nodes. The PVC supported release is available via: frameworks/2022.10.15.001 which supports implicit scaling frameworks/2022.10.15.002 which supports explicit scaling This version of the frameworks uses the intel_compute_runtime/release/agama-prerelease-519 modulefile. Because of this, the performance of the NRE models is not as good as what can be achieved with later versions of the driver used by the intel_compute_runtime/release/pvc-prq-17 or intel_compute_runtime/release/pvc-prq-29 modulefiles. More effort is needed in order to provide frameworks modulefiles that can support these later versions of the gpu drivers on the florentia nodes in the jlse, so at this time we wanted to provide NRE model functionality on the florentia nodes. There are additional updates expected in the coming weeks. Additional information on setting up and working with this update can be found here for pytorch: https://wiki.jlse.anl.gov/display/inteldga/2022.3.0+-+Pytorch and here for tensorflow: https://wiki.jlse.anl.gov/display/inteldga/2022.3.0+-+Tensorflow Please let us know of any issues encountered. Rebecca Ramer rebecca.ramer(a)intel.com<mailto:[email protected]>

1 0