July 2023 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

New Kokkos and Cabana modules and default on sunspot
by Williams, Timothy J. 28 Jul '23

28 Jul '23

On sunspot, new versions of Kokkos and Cabana were built against the two new oneAPI SDK versions. The defaults you get from module load kokkos cabana are now the ones built against oneapi/eng-compiler/2023.05.15.006 (the new default oneAPI version). This update has not yet been made on JLSE, and won’t be until some time after the weekend shutdown. Best regards, Tim ------- Timothy J. Williams, Ph.D. --- Argonne National Laboratory --- tjwilliams(a)anl.gov -------

1 0

Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions
by Chan-nui, Christopher 27 Jul '23

27 Jul '23

I have been informed that “This warning is not a problem. It is just an indication that the ZEinfo version you are using with this driver was not tested” and that the warning will be removed in a future release. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: John Mellor-Crummey <johnmc(a)rice.edu> Date: Wednesday, July 26, 2023 at 8:22 PM To: Bertoni, Colleen <bertoni(a)anl.gov>, Chan-nui, Christopher <christopher.chan-nui(a)intel.com> Cc: Mellor-Crummey, John <johnmc(a)rice.edu>, ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions Colleen and Christopher, I’ll note that GTPin isn’t part of the installation of the oneapi environment. Using gtpin-3.5 (NDA) obtained from the GTPin team with the new environment, GTPin prints the following when used from HPCToolkit: GTPin WARNING: Newer ZEInfo version detected: '1.28' That seems concerning, though it still seems to work. -- John Mellor-Crummey Professor Dept of Computer Science Rice University email: johnmc(a)rice.edu<mailto:[email protected]> phone: 713-348-5179 On Jul 26, 2023, at 8:49 AM, Bertoni, Colleen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: Thanks Chris! I put three sections below, "About oneapi/eng-compiler/2023.05.15.006", "About oneapi/eng-compiler/2023.05.15.007", and "Workarounds and caveats". ## About oneapi/eng-compiler/2023.05.15.006 Like Chris said, the new default (oneapi/eng-compiler/2023.05.15.006) is the same as 2023.05.15.003 release with the runtime updated to agama-devel-647, so the bug fixes/regressions compared to 2022.12.30.003 (the current default) are similar to the previous email in this chain. However, there were 14 new fixes thanks to the UMD upgrade: ``` > ../ath.py compare --list --status=fixed --new-file reproducers_2023-Jul-25_output_eng-compiler/2023.05.15.006/output.json --reference-file ../standard/2023.05.15.003/reproducers/output.json source/reproducers/dpcpp/events_not_destroy [CMPLRLLVM-28962,CMPLRLLVM-29331] source/reproducers/dpcpp/pci_bidirectional:pci_bidirectional_ath_compile [XDEPS-3128] source/reproducers/dpcpp/pci_device_vs_shared:pci_device_vs_shared_ath_run source/reproducers/dpcpp/sycl_platform_default_context [CMPLRLLVM-38150] source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/unexpected_kernel_size [CMPLRLLVM-44281] source/reproducers/hybrid/oversubscribing:oversubscribing_more_shared_memory_wa_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_implicit_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_one_tile_ath_run [GSD-786] source/reproducers/hybrid/pci_bw:pci_bw_ath_run [CMPLRLLVM-44164,XDEPS-5855] source/reproducers/l0/cache:cache_persistant_cache_ath_run source/reproducers/mpi/device_mpi [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/device_mpi:device_mpi_immediate_command [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/mpi_gatherv:mpi_gatherv_ath_compile ``` ## About oneapi/eng-compiler/2023.05.15.007 The new non-default SDK “oneapi/eng-compiler/2023.05.15.007” has agama-647, a new compiler, MKL, and a new mpich. There are 17 fixes and 16 regressions compared to 2023.05.15.006 (listed below), although many are from the same root issue, so there are less real regressions than 16. Additionally, some notable results are that there were performance improvements for batched getrs (from gtensor/GENE and Strumpack ECP) and and non-power-of-2 FFTs (from HeFFTe). ## Workarounds and caveats There is a known issue with OpenMP offload and mpich/52.2 which results in a segfault at the end of a run. Workarounds for this can be: 1. Turn off GPU-aware MPI (export MPIR_CVAR_ENABLE_GPU=0) 2. Turn off Immediate command lists for OpenMP offload (export LIBOMPTARGET_LEVEL_ZERO_USE_IMMEDIATE_COMMAND_LIST=0) 3. Downgrade to mpich/51.2 The reason mpich/52.2 is loaded is that it fixes an incorrect answer issue with MPI I/O calls, as seen in parallel HDF5. This also can result in a “free(): double free detected in tcache 2” error on multiple nodes, and a workaround for that is ``` unset MPIR_CVAR_CH4_COLL_SELECTION_TUNING_JSON_FILE unset MPIR_CVAR_COLL_SELECTION_TUNING_JSON_FILE unset MPIR_CVAR_CH4_POSIX_COLL_SELECTION_TUNING_JSON_FILE``` # Fixes for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 ``` source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_in_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_out_of_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/ifx/atomic_update_ice source/reproducers/mkl/minimal_zgeqrf_tester:minimal_zgeqrf_tester_ze00_ath_run source/reproducers/mkl/singular_should_throw:singular_should_throw_opencl [MKLD-14659] source/reproducers/mkl/slow_batch_getrf:slow_batch_getrf_use_malloc_host [MKLD-15077,MKLD-15079] source/reproducers/mkl/zgemm source/reproducers/mpi/Testmpio:Testmpio_ath_run [HPCS-8110] source/reproducers/mpi/hacc_sunspot-gio source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_t_pmulti_dset_ath_run [HPCS-8110] source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_testphdf5_ath_run [HPCS-8110] source/reproducers/openmp/findclus:findclus_ath_run [GSD-5335] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20 [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20_reduction [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_simple_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/reduction_complex_perf:reduction_complex_perf_ath_run [CMPLRLLVM-37471,CMPLRLLVM-38249,CMPLRLLVM-42440] P0 ``` # Regressions for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 ``` source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_opcode source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_operand_syntax source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_duplicate_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_illegal_exec_size source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_region source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_simple source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_decl source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_pred source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_wrong_declare source/reproducers/dpcpp/multigpu_multiqueue source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/vecadd_v0_multidevice source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 [CMPLRLLVM-39495] source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 [CMPLRLLVM-39495] source/reproducers/mpi/openmp_target_zam:openmp_target_zam_zam1_ath_run [HPCS-8169] ``` ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Tuesday, July 25, 2023 4:58 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions An updated version of the Aurora SDK is now available and will be made default Wednesday at 12pm CDT. The module files for this release are: oneapi/eng-compiler/2023.05.15.006 – This will be made default, it is the 2023.05.15.003 release with the UMD updated to 647 oneapi/eng-compiler/2023.05.15.007 – Updated engineering compiler/mkl/mpich + UMD 647 This is based on the 2023.1 release of the oneAPI SDK. Colleen will provide caveats, workarounds and other useful information about these releases shortly. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]> From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Thursday, June 22, 2023 at 1:26 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions A new version of the Aurora SDK is now available. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release are: oneapi/release/2023.05.15.001 oneapi/eng-compiler/2023.05.15.003 This is based on the 2023.1 release of the oneAPI SDK. Release notes are available at https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… Addendums for the engineering components are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]> -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions
by John Mellor-Crummey 26 Jul '23

26 Jul '23

Colleen and Christopher, I’ll note that GTPin isn’t part of the installation of the oneapi environment. Using gtpin-3.5 (NDA) obtained from the GTPin team with the new environment, GTPin prints the following when used from HPCToolkit: GTPin WARNING: Newer ZEInfo version detected: '1.28' That seems concerning, though it still seems to work. -- John Mellor-Crummey Professor Dept of Computer Science Rice University email: johnmc(a)rice.edu phone: 713-348-5179 > On Jul 26, 2023, at 8:49 AM, Bertoni, Colleen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> wrote: > > Thanks Chris! I put three sections below, "About oneapi/eng-compiler/2023.05.15.006", "About oneapi/eng-compiler/2023.05.15.007", and "Workarounds and caveats". > > ## About oneapi/eng-compiler/2023.05.15.006 > > Like Chris said, the new default (oneapi/eng-compiler/2023.05.15.006) is the same as 2023.05.15.003 release with the runtime updated to agama-devel-647, so the bug fixes/regressions compared to 2022.12.30.003 (the current default) are similar to the previous email in this chain. However, there were 14 new fixes thanks to the UMD upgrade: > > ``` > > ../ath.py compare --list --status=fixed --new-file reproducers_2023-Jul-25_output_eng-compiler/2023.05.15.006/output.json --reference-file ../standard/2023.05.15.003/reproducers/output.json > source/reproducers/dpcpp/events_not_destroy [CMPLRLLVM-28962,CMPLRLLVM-29331] > source/reproducers/dpcpp/pci_bidirectional:pci_bidirectional_ath_compile [XDEPS-3128] > source/reproducers/dpcpp/pci_device_vs_shared:pci_device_vs_shared_ath_run > source/reproducers/dpcpp/sycl_platform_default_context [CMPLRLLVM-38150] > source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] > source/reproducers/dpcpp/unexpected_kernel_size [CMPLRLLVM-44281] > source/reproducers/hybrid/oversubscribing:oversubscribing_more_shared_memory_wa_ath_run [GSD-786] > source/reproducers/hybrid/oversubscribing:oversubscribing_omp_implicit_ath_run [GSD-786] > source/reproducers/hybrid/oversubscribing:oversubscribing_omp_one_tile_ath_run [GSD-786] > source/reproducers/hybrid/pci_bw:pci_bw_ath_run [CMPLRLLVM-44164,XDEPS-5855] > source/reproducers/l0/cache:cache_persistant_cache_ath_run > source/reproducers/mpi/device_mpi [GSD-4488,HPCS-7809,HPCS-8169] > source/reproducers/mpi/device_mpi:device_mpi_immediate_command [GSD-4488,HPCS-7809,HPCS-8169] > source/reproducers/mpi/mpi_gatherv:mpi_gatherv_ath_compile > ``` > > ## About oneapi/eng-compiler/2023.05.15.007 > > The new non-default SDK “oneapi/eng-compiler/2023.05.15.007” has agama-647, a new compiler, MKL, and a new mpich. There are 17 fixes and 16 regressions compared to 2023.05.15.006 (listed below), although many are from the same root issue, so there are less real regressions than 16. > > Additionally, some notable results are that there were performance improvements for batched getrs (from gtensor/GENE and Strumpack ECP) and and non-power-of-2 FFTs (from HeFFTe). > > ## Workarounds and caveats > > There is a known issue with OpenMP offload and mpich/52.2 which results in a segfault at the end of a run. Workarounds for this can be: > > 1. Turn off GPU-aware MPI (export MPIR_CVAR_ENABLE_GPU=0) > 2. Turn off Immediate command lists for OpenMP offload (export LIBOMPTARGET_LEVEL_ZERO_USE_IMMEDIATE_COMMAND_LIST=0) > 3. Downgrade to mpich/51.2 > > The reason mpich/52.2 is loaded is that it fixes an incorrect answer issue with MPI I/O calls, as seen in parallel HDF5. This also can result in a “free(): double free detected in tcache 2” error on multiple nodes, and a workaround for that is > ``` unset MPIR_CVAR_CH4_COLL_SELECTION_TUNING_JSON_FILE > unset MPIR_CVAR_COLL_SELECTION_TUNING_JSON_FILE > unset MPIR_CVAR_CH4_POSIX_COLL_SELECTION_TUNING_JSON_FILE``` > > > # Fixes for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 > > ``` > source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_in_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 > source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_out_of_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 > source/reproducers/ifx/atomic_update_ice > source/reproducers/mkl/minimal_zgeqrf_tester:minimal_zgeqrf_tester_ze00_ath_run > source/reproducers/mkl/singular_should_throw:singular_should_throw_opencl [MKLD-14659] > source/reproducers/mkl/slow_batch_getrf:slow_batch_getrf_use_malloc_host [MKLD-15077,MKLD-15079] > source/reproducers/mkl/zgemm > source/reproducers/mpi/Testmpio:Testmpio_ath_run [HPCS-8110] > source/reproducers/mpi/hacc_sunspot-gio > source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_t_pmulti_dset_ath_run [HPCS-8110] > source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_testphdf5_ath_run [HPCS-8110] > source/reproducers/openmp/findclus:findclus_ath_run [GSD-5335] > source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20 [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] > source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20_reduction [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] > source/reproducers/openmp/map_to_many_sync:map_to_many_sync_ath_run [CMPLRLLVM-32920] > source/reproducers/openmp/map_to_many_sync:map_to_many_sync_simple_ath_run [CMPLRLLVM-32920] > source/reproducers/openmp/reduction_complex_perf:reduction_complex_perf_ath_run [CMPLRLLVM-37471,CMPLRLLVM-38249,CMPLRLLVM-42440] P0 > ``` > > # Regressions for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 > > ``` > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_opcode > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_operand_syntax > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_duplicate_label > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_illegal_exec_size > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_label > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_region > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_simple > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_decl > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_pred > source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_wrong_declare > source/reproducers/dpcpp/multigpu_multiqueue > source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] > source/reproducers/dpcpp/vecadd_v0_multidevice > source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 [CMPLRLLVM-39495] > source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 [CMPLRLLVM-39495] > source/reproducers/mpi/openmp_target_zam:openmp_target_zam_zam1_ath_run [HPCS-8169] > ``` > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov <mailto:[email protected]>> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]>> > Sent: Tuesday, July 25, 2023 4:58 PM > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]> <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]>> > Subject: Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions > > An updated version of the Aurora SDK is now available and will be made default Wednesday at 12pm CDT. > > The module files for this release are: > > oneapi/eng-compiler/2023.05.15.006 – This will be made default, it is the 2023.05.15.003 release with the UMD updated to 647 > oneapi/eng-compiler/2023.05.15.007 – Updated engineering compiler/mkl/mpich + UMD 647 > > This is based on the 2023.1 release of the oneAPI SDK. > > Colleen will provide caveats, workarounds and other useful information about these releases shortly. > > -- > Christopher Chan-Nui > christopher.chan-nui(a)intel.com <mailto:[email protected]> > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov <mailto:[email protected]>> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]>> > Date: Thursday, June 22, 2023 at 1:26 PM > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]> <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]>> > Subject: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions > > A new version of the Aurora SDK is now available. This will NOT be enabled as the default version. Running jobs will NOT be affected. > > The module files for this release are: > > oneapi/release/2023.05.15.001 > oneapi/eng-compiler/2023.05.15.003 > > This is based on the 2023.1 release of the oneAPI SDK. > > Release notes are available at > https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… <https://www.intel.com/content/www/us/en/developer/articles/release-notes/in…> > https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… <https://www.intel.com/content/www/us/en/developer/articles/release-notes/in…> > > > Addendums for the engineering components are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes <https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes>. > > > -- > Christopher Chan-Nui > christopher.chan-nui(a)intel.com <mailto:[email protected]> > > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]> > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss <https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss>

1 0

Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions
by Chan-nui, Christopher 26 Jul '23

26 Jul '23

The default oneAPI modules on Sunspot and JLSE have been updated. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Bertoni, Colleen <bertoni(a)anl.gov> Date: Wednesday, July 26, 2023 at 8:50 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>, Chan-nui, Christopher <christopher.chan-nui(a)intel.com> Subject: Re: New oneAPI release and eng-compiler versions Thanks Chris! I put three sections below, "About oneapi/eng-compiler/2023.05.15.006", "About oneapi/eng-compiler/2023.05.15.007", and "Workarounds and caveats". ## About oneapi/eng-compiler/2023.05.15.006 Like Chris said, the new default (oneapi/eng-compiler/2023.05.15.006) is the same as 2023.05.15.003 release with the runtime updated to agama-devel-647, so the bug fixes/regressions compared to 2022.12.30.003 (the current default) are similar to the previous email in this chain. However, there were 14 new fixes thanks to the UMD upgrade: ``` > ../ath.py compare --list --status=fixed --new-file reproducers_2023-Jul-25_output_eng-compiler/2023.05.15.006/output.json --reference-file ../standard/2023.05.15.003/reproducers/output.json source/reproducers/dpcpp/events_not_destroy [CMPLRLLVM-28962,CMPLRLLVM-29331] source/reproducers/dpcpp/pci_bidirectional:pci_bidirectional_ath_compile [XDEPS-3128] source/reproducers/dpcpp/pci_device_vs_shared:pci_device_vs_shared_ath_run source/reproducers/dpcpp/sycl_platform_default_context [CMPLRLLVM-38150] source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/unexpected_kernel_size [CMPLRLLVM-44281] source/reproducers/hybrid/oversubscribing:oversubscribing_more_shared_memory_wa_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_implicit_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_one_tile_ath_run [GSD-786] source/reproducers/hybrid/pci_bw:pci_bw_ath_run [CMPLRLLVM-44164,XDEPS-5855] source/reproducers/l0/cache:cache_persistant_cache_ath_run source/reproducers/mpi/device_mpi [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/device_mpi:device_mpi_immediate_command [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/mpi_gatherv:mpi_gatherv_ath_compile ``` ## About oneapi/eng-compiler/2023.05.15.007 The new non-default SDK “oneapi/eng-compiler/2023.05.15.007” has agama-647, a new compiler, MKL, and a new mpich. There are 17 fixes and 16 regressions compared to 2023.05.15.006 (listed below), although many are from the same root issue, so there are less real regressions than 16. Additionally, some notable results are that there were performance improvements for batched getrs (from gtensor/GENE and Strumpack ECP) and and non-power-of-2 FFTs (from HeFFTe). ## Workarounds and caveats There is a known issue with OpenMP offload and mpich/52.2 which results in a segfault at the end of a run. Workarounds for this can be: 1. Turn off GPU-aware MPI (export MPIR_CVAR_ENABLE_GPU=0) 2. Turn off Immediate command lists for OpenMP offload (export LIBOMPTARGET_LEVEL_ZERO_USE_IMMEDIATE_COMMAND_LIST=0) 3. Downgrade to mpich/51.2 The reason mpich/52.2 is loaded is that it fixes an incorrect answer issue with MPI I/O calls, as seen in parallel HDF5. This also can result in a “free(): double free detected in tcache 2” error on multiple nodes, and a workaround for that is ``` unset MPIR_CVAR_CH4_COLL_SELECTION_TUNING_JSON_FILE unset MPIR_CVAR_COLL_SELECTION_TUNING_JSON_FILE unset MPIR_CVAR_CH4_POSIX_COLL_SELECTION_TUNING_JSON_FILE``` # Fixes for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 ``` source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_in_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_out_of_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/ifx/atomic_update_ice source/reproducers/mkl/minimal_zgeqrf_tester:minimal_zgeqrf_tester_ze00_ath_run source/reproducers/mkl/singular_should_throw:singular_should_throw_opencl [MKLD-14659] source/reproducers/mkl/slow_batch_getrf:slow_batch_getrf_use_malloc_host [MKLD-15077,MKLD-15079] source/reproducers/mkl/zgemm source/reproducers/mpi/Testmpio:Testmpio_ath_run [HPCS-8110] source/reproducers/mpi/hacc_sunspot-gio source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_t_pmulti_dset_ath_run [HPCS-8110] source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_testphdf5_ath_run [HPCS-8110] source/reproducers/openmp/findclus:findclus_ath_run [GSD-5335] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20 [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20_reduction [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_simple_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/reduction_complex_perf:reduction_complex_perf_ath_run [CMPLRLLVM-37471,CMPLRLLVM-38249,CMPLRLLVM-42440] P0 ``` # Regressions for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 ``` source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_opcode source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_operand_syntax source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_duplicate_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_illegal_exec_size source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_region source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_simple source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_decl source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_pred source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_wrong_declare source/reproducers/dpcpp/multigpu_multiqueue source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/vecadd_v0_multidevice source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 [CMPLRLLVM-39495] source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 [CMPLRLLVM-39495] source/reproducers/mpi/openmp_target_zam:openmp_target_zam_zam1_ath_run [HPCS-8169] ``` ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, July 25, 2023 4:58 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions An updated version of the Aurora SDK is now available and will be made default Wednesday at 12pm CDT. The module files for this release are: oneapi/eng-compiler/2023.05.15.006 – This will be made default, it is the 2023.05.15.003 release with the UMD updated to 647 oneapi/eng-compiler/2023.05.15.007 – Updated engineering compiler/mkl/mpich + UMD 647 This is based on the 2023.1 release of the oneAPI SDK. Colleen will provide caveats, workarounds and other useful information about these releases shortly. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Thursday, June 22, 2023 at 1:26 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions A new version of the Aurora SDK is now available. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release are: oneapi/release/2023.05.15.001 oneapi/eng-compiler/2023.05.15.003 This is based on the 2023.1 release of the oneAPI SDK. Release notes are available at https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… Addendums for the engineering components are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions
by Bertoni, Colleen 26 Jul '23

26 Jul '23

Thanks Chris! I put three sections below, "About oneapi/eng-compiler/2023.05.15.006", "About oneapi/eng-compiler/2023.05.15.007", and "Workarounds and caveats". ## About oneapi/eng-compiler/2023.05.15.006 Like Chris said, the new default (oneapi/eng-compiler/2023.05.15.006) is the same as 2023.05.15.003 release with the runtime updated to agama-devel-647, so the bug fixes/regressions compared to 2022.12.30.003 (the current default) are similar to the previous email in this chain. However, there were 14 new fixes thanks to the UMD upgrade: ``` > ../ath.py compare --list --status=fixed --new-file reproducers_2023-Jul-25_output_eng-compiler/2023.05.15.006/output.json --reference-file ../standard/2023.05.15.003/reproducers/output.json source/reproducers/dpcpp/events_not_destroy [CMPLRLLVM-28962,CMPLRLLVM-29331] source/reproducers/dpcpp/pci_bidirectional:pci_bidirectional_ath_compile [XDEPS-3128] source/reproducers/dpcpp/pci_device_vs_shared:pci_device_vs_shared_ath_run source/reproducers/dpcpp/sycl_platform_default_context [CMPLRLLVM-38150] source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/unexpected_kernel_size [CMPLRLLVM-44281] source/reproducers/hybrid/oversubscribing:oversubscribing_more_shared_memory_wa_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_implicit_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_one_tile_ath_run [GSD-786] source/reproducers/hybrid/pci_bw:pci_bw_ath_run [CMPLRLLVM-44164,XDEPS-5855] source/reproducers/l0/cache:cache_persistant_cache_ath_run source/reproducers/mpi/device_mpi [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/device_mpi:device_mpi_immediate_command [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/mpi_gatherv:mpi_gatherv_ath_compile ``` ## About oneapi/eng-compiler/2023.05.15.007 The new non-default SDK “oneapi/eng-compiler/2023.05.15.007” has agama-647, a new compiler, MKL, and a new mpich. There are 17 fixes and 16 regressions compared to 2023.05.15.006 (listed below), although many are from the same root issue, so there are less real regressions than 16. Additionally, some notable results are that there were performance improvements for batched getrs (from gtensor/GENE and Strumpack ECP) and and non-power-of-2 FFTs (from HeFFTe). ## Workarounds and caveats There is a known issue with OpenMP offload and mpich/52.2 which results in a segfault at the end of a run. Workarounds for this can be: 1. Turn off GPU-aware MPI (export MPIR_CVAR_ENABLE_GPU=0) 2. Turn off Immediate command lists for OpenMP offload (export LIBOMPTARGET_LEVEL_ZERO_USE_IMMEDIATE_COMMAND_LIST=0) 3. Downgrade to mpich/51.2 The reason mpich/52.2 is loaded is that it fixes an incorrect answer issue with MPI I/O calls, as seen in parallel HDF5. This also can result in a “free(): double free detected in tcache 2” error on multiple nodes, and a workaround for that is ``` unset MPIR_CVAR_CH4_COLL_SELECTION_TUNING_JSON_FILE unset MPIR_CVAR_COLL_SELECTION_TUNING_JSON_FILE unset MPIR_CVAR_CH4_POSIX_COLL_SELECTION_TUNING_JSON_FILE``` # Fixes for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 ``` source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_in_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_out_of_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/ifx/atomic_update_ice source/reproducers/mkl/minimal_zgeqrf_tester:minimal_zgeqrf_tester_ze00_ath_run source/reproducers/mkl/singular_should_throw:singular_should_throw_opencl [MKLD-14659] source/reproducers/mkl/slow_batch_getrf:slow_batch_getrf_use_malloc_host [MKLD-15077,MKLD-15079] source/reproducers/mkl/zgemm source/reproducers/mpi/Testmpio:Testmpio_ath_run [HPCS-8110] source/reproducers/mpi/hacc_sunspot-gio source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_t_pmulti_dset_ath_run [HPCS-8110] source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_testphdf5_ath_run [HPCS-8110] source/reproducers/openmp/findclus:findclus_ath_run [GSD-5335] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20 [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20_reduction [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_simple_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/reduction_complex_perf:reduction_complex_perf_ath_run [CMPLRLLVM-37471,CMPLRLLVM-38249,CMPLRLLVM-42440] P0 ``` # Regressions for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 ``` source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_opcode source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_operand_syntax source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_duplicate_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_illegal_exec_size source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_region source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_simple source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_decl source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_pred source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_wrong_declare source/reproducers/dpcpp/multigpu_multiqueue source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/vecadd_v0_multidevice source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 [CMPLRLLVM-39495] source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 [CMPLRLLVM-39495] source/reproducers/mpi/openmp_target_zam:openmp_target_zam_zam1_ath_run [HPCS-8169] ``` ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, July 25, 2023 4:58 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions An updated version of the Aurora SDK is now available and will be made default Wednesday at 12pm CDT. The module files for this release are: oneapi/eng-compiler/2023.05.15.006 – This will be made default, it is the 2023.05.15.003 release with the UMD updated to 647 oneapi/eng-compiler/2023.05.15.007 – Updated engineering compiler/mkl/mpich + UMD 647 This is based on the 2023.1 release of the oneAPI SDK. Colleen will provide caveats, workarounds and other useful information about these releases shortly. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Thursday, June 22, 2023 at 1:26 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions A new version of the Aurora SDK is now available. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release are: oneapi/release/2023.05.15.001 oneapi/eng-compiler/2023.05.15.003 This is based on the 2023.1 release of the oneAPI SDK. Release notes are available at https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… Addendums for the engineering components are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions
by Chan-nui, Christopher 25 Jul '23

25 Jul '23

An updated version of the Aurora SDK is now available and will be made default Wednesday at 12pm CDT. The module files for this release are: oneapi/eng-compiler/2023.05.15.006 – This will be made default, it is the 2023.05.15.003 release with the UMD updated to 647 oneapi/eng-compiler/2023.05.15.007 – Updated engineering compiler/mkl/mpich + UMD 647 This is based on the 2023.1 release of the oneAPI SDK. Colleen will provide caveats, workarounds and other useful information about these releases shortly. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Thursday, June 22, 2023 at 1:26 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions A new version of the Aurora SDK is now available. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release are: oneapi/release/2023.05.15.001 oneapi/eng-compiler/2023.05.15.003 This is based on the 2023.1 release of the oneAPI SDK. Release notes are available at https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… https://www.intel.com/content/www/us/en/developer/articles/release-notes/in… Addendums for the engineering components are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] Unexpected behavior when checking for _OPENMP on Sunspot
by Steinbrecher, Patrick 12 Jul '23

12 Jul '23

There are two passes in compiler when you compile with -fiopenmp and -fsycl. One pass is for SYCL, so you have to make your code legal to be compiled without OpenMP defined. You have to add ifdefs via __SYCL_DEVICE_ONLY__ macro to make code legal for SYCL pass. From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Asim YarKhan via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Wednesday, July 12, 2023 at 11:15 To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Unexpected behavior when checking for _OPENMP on Sunspot Hi, I am having some unexpected behaviour when I am checking the _OPENMP compiler defined variable (as part of our SLATE linear algebra library). Sometimes _OPENMP is not set as expected. I have a minimal 26-line example that shows this behavior. The comments in the top of the file describe building and the observed behavior. Please let me know what I may be doing incorrectly here. Thanks, Asim YarKhan // Sometimes the icpx compiler does not seem to set the _OPENMP // compiler define. // Desired output from this test: "_OPENMP is defined" // Works: icpx -fiopenmp -o openmp_test.o -c openmp_test.cc // Fails: icpx -fsycl -fiopenmp -std=c++17 -o openmp_test.o -c openmp_test.cc // icpx -fsycl -fiopenmp -std=c++17 -o openmp_test.o -c openmp_test.cc // Output on sunspot // openmp_test.cc:26:2: error: _OPENMP is NOT defined // #error _OPENMP is NOT defined // ^ // 1 error generated.

1 0

Unexpected behavior when checking for _OPENMP on Sunspot
by Asim YarKhan 12 Jul '23

12 Jul '23

Hi, I am having some unexpected behaviour when I am checking the _OPENMP compiler defined variable (as part of our SLATE linear algebra library). Sometimes _OPENMP is not set as expected. I have a minimal 26-line example that shows this behavior. The comments in the top of the file describe building and the observed behavior. Please let me know what I may be doing incorrectly here. Thanks, Asim YarKhan // Sometimes the icpx compiler does not seem to set the _OPENMP // compiler define. // Desired output from this test: "_OPENMP is defined" // Works: icpx -fiopenmp -o openmp_test.o -c openmp_test.cc // Fails: icpx -fsycl -fiopenmp -std=c++17 -o openmp_test.o -c openmp_test.cc // icpx -fsycl -fiopenmp -std=c++17 -o openmp_test.o -c openmp_test.cc // Output on sunspot // openmp_test.cc:26:2: error: _OPENMP is NOT defined // #error _OPENMP is NOT defined // ^ // 1 error generated.

1 0

Re: [Ecp-aurora-sdk-discuss] Test failures with differing behavior using 1 tile or 2 on Sunspot
by Applencourt, Thomas 10 Jul '23

10 Jul '23

Hi Natalie, We were able to write a minimal reproducer (I will send it to you in another email). The bug is in `oneapi::zgemm`. We will report it to intel. Thanks you for the initial reproducer, it helped a lot, Regards, Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1122 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Bertoni, Colleen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Friday, July 7, 2023 10:28 To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Natalie Beams <nbeams(a)icl.utk.edu> Subject: Re: [Ecp-aurora-sdk-discuss] Test failures with differing behavior using 1 tile or 2 on Sunspot Thanks a lot for the report Natalie! I'll try to reproduce and then we can send a bug report to Intel. I'll let you know once I can reproduce! Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Natalie Beams via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, July 6, 2023 7:04 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Test failures with differing behavior using 1 tile or 2 on Sunspot Hello, I have been plagued by a mysterious error in the ported-to-SYCL MAGMA code for QR with double precision complex values (zgeqrf). I have attached what I hope should be all required files for a reproducer. It can be compiled with a simple `make` and tested with: ./testing_zgeqrf -M [rows in matrix] -nb [block size] (can also use -N for columns in matrix, but if it's not set, it will default to M = N) Most recently I have been using module load oneapi/eng-compiler/2023.05.15.003 when testing on Sunspot. Tests using an older compiler module and a different 2D memcopy routine (the current code uses Intel's new-ish extension to replace the matrix mem copy from dpct) showed the same behavior. It has been simplified a lot compared to the original ported MAGMA code, but unfortunately, I cannot figure out how to simplify it further and still reproduce the issue. The behavior is dependent on whether I use one tile (ZE_AFFINITY_MASK=0.0) or two (just setting the queue to use device 0, with ZE_AFFINITY_MASK not set). For example, the size at which the test starts to fail (M >= {size} fails, < is ok): | one tile | two tiles =================================== nb = 32 | 1217 | 1473 nb = 64 | 1281 | 1537 nb = 128 | ? (maybe never?) | 1665 If I comment out the call to trmm and the second gemm in magma_zlarfb_gpu, the output from one and two tiles appears to be the same (though they are both incorrect, of course). But my attempts at reproducing the issue with only trmm or gemm in standalone testers have not been successful. So I'm not sure whether the issue is really with MKL or not. This simplified code focuses only on double precision complex, but in the original MAGMA test, I also tested double precision real (which was fine for all sizes tested) as well as single precision complex (also fine). Any help narrowing down what might be going on is greatly appreciated. Thanks, Natalie Beams

1 0

Re: [Ecp-aurora-sdk-discuss] Test failures with differing behavior using 1 tile or 2 on Sunspot
by Bertoni, Colleen 07 Jul '23

07 Jul '23

Thanks a lot for the report Natalie! I'll try to reproduce and then we can send a bug report to Intel. I'll let you know once I can reproduce! Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Natalie Beams via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, July 6, 2023 7:04 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Test failures with differing behavior using 1 tile or 2 on Sunspot Hello, I have been plagued by a mysterious error in the ported-to-SYCL MAGMA code for QR with double precision complex values (zgeqrf). I have attached what I hope should be all required files for a reproducer. It can be compiled with a simple `make` and tested with: ./testing_zgeqrf -M [rows in matrix] -nb [block size] (can also use -N for columns in matrix, but if it's not set, it will default to M = N) Most recently I have been using module load oneapi/eng-compiler/2023.05.15.003 when testing on Sunspot. Tests using an older compiler module and a different 2D memcopy routine (the current code uses Intel's new-ish extension to replace the matrix mem copy from dpct) showed the same behavior. It has been simplified a lot compared to the original ported MAGMA code, but unfortunately, I cannot figure out how to simplify it further and still reproduce the issue. The behavior is dependent on whether I use one tile (ZE_AFFINITY_MASK=0.0) or two (just setting the queue to use device 0, with ZE_AFFINITY_MASK not set). For example, the size at which the test starts to fail (M >= {size} fails, < is ok): | one tile | two tiles =================================== nb = 32 | 1217 | 1473 nb = 64 | 1281 | 1537 nb = 128 | ? (maybe never?) | 1665 If I comment out the call to trmm and the second gemm in magma_zlarfb_gpu, the output from one and two tiles appears to be the same (though they are both incorrect, of course). But my attempts at reproducing the issue with only trmm or gemm in standalone testers have not been successful. So I'm not sure whether the issue is really with MKL or not. This simplified code focuses only on double precision complex, but in the original MAGMA test, I also tested double precision real (which was fine for all sizes tested) as well as single precision complex (also fine). Any help narrowing down what might be going on is greatly appreciated. Thanks, Natalie Beams

1 0