The default oneAPI modules on Sunspot and JLSE have been updated. -- Christopher Chan-Nui [email protected] From: Bertoni, Colleen <[email protected]> Date: Wednesday, July 26, 2023 at 8:50 AM To: [email protected] <[email protected]>, Chan-nui, Christopher <[email protected]> Subject: Re: New oneAPI release and eng-compiler versions Thanks Chris! I put three sections below, "About oneapi/eng-compiler/2023.05.15.006", "About oneapi/eng-compiler/2023.05.15.007", and "Workarounds and caveats". ## About oneapi/eng-compiler/2023.05.15.006 Like Chris said, the new default (oneapi/eng-compiler/2023.05.15.006) is the same as 2023.05.15.003 release with the runtime updated to agama-devel-647, so the bug fixes/regressions compared to 2022.12.30.003 (the current default) are similar to the previous email in this chain. However, there were 14 new fixes thanks to the UMD upgrade: ```

../ath.py compare --list --status=fixed --new-file reproducers_2023-Jul-25_output_eng-compiler/2023.05.15.006/output.json --reference-file ../standard/2023.05.15.003/reproducers/output.json source/reproducers/dpcpp/events_not_destroy [CMPLRLLVM-28962,CMPLRLLVM-29331] source/reproducers/dpcpp/pci_bidirectional:pci_bidirectional_ath_compile [XDEPS-3128] source/reproducers/dpcpp/pci_device_vs_shared:pci_device_vs_shared_ath_run source/reproducers/dpcpp/sycl_platform_default_context [CMPLRLLVM-38150] source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/unexpected_kernel_size [CMPLRLLVM-44281] source/reproducers/hybrid/oversubscribing:oversubscribing_more_shared_memory_wa_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_implicit_ath_run [GSD-786] source/reproducers/hybrid/oversubscribing:oversubscribing_omp_one_tile_ath_run [GSD-786] source/reproducers/hybrid/pci_bw:pci_bw_ath_run [CMPLRLLVM-44164,XDEPS-5855] source/reproducers/l0/cache:cache_persistant_cache_ath_run source/reproducers/mpi/device_mpi [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/device_mpi:device_mpi_immediate_command [GSD-4488,HPCS-7809,HPCS-8169] source/reproducers/mpi/mpi_gatherv:mpi_gatherv_ath_compile

## About oneapi/eng-compiler/2023.05.15.007

The new non-default SDK “oneapi/eng-compiler/2023.05.15.007” has agama-647, a new compiler, MKL, and a new mpich. There are 17 fixes and 16 regressions compared to 2023.05.15.006 (listed below), although many are from the same root issue, so there are less real regressions than 16.

Additionally, some notable results are that there were performance improvements for batched getrs (from  gtensor/GENE and Strumpack ECP) and and non-power-of-2 FFTs (from HeFFTe).

## Workarounds and caveats

There is a known issue with OpenMP offload and mpich/52.2 which results in a segfault at the end of a run. Workarounds for this can be:

1. Turn off GPU-aware MPI (export MPIR_CVAR_ENABLE_GPU=0)
2. Turn off Immediate command lists for OpenMP offload (export LIBOMPTARGET_LEVEL_ZERO_USE_IMMEDIATE_COMMAND_LIST=0)
3. Downgrade to mpich/51.2

The reason mpich/52.2 is loaded is that it fixes an incorrect answer issue with MPI I/O calls, as seen in parallel HDF5. This also can result in a “free(): double free detected in tcache 2” error on multiple nodes, and a workaround for that is
``` unset MPIR_CVAR_CH4_COLL_SELECTION_TUNING_JSON_FILE
unset MPIR_CVAR_COLL_SELECTION_TUNING_JSON_FILE
unset MPIR_CVAR_CH4_POSIX_COLL_SELECTION_TUNING_JSON_FILE```


# Fixes for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006

source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_in_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/hybrid/oneConcurency:oneConcurency_sycl_con_out_of_order_ath_run [CMPLRLLVM-34779,CMPLRLLVM-38250,CMPLRLLVM-40729,XDEPS-2202,XDEPS-3493,XDEPS-5689] P0 source/reproducers/ifx/atomic_update_ice source/reproducers/mkl/minimal_zgeqrf_tester:minimal_zgeqrf_tester_ze00_ath_run source/reproducers/mkl/singular_should_throw:singular_should_throw_opencl [MKLD-14659] source/reproducers/mkl/slow_batch_getrf:slow_batch_getrf_use_malloc_host [MKLD-15077,MKLD-15079] source/reproducers/mkl/zgemm source/reproducers/mpi/Testmpio:Testmpio_ath_run [HPCS-8110] source/reproducers/mpi/hacc_sunspot-gio source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_t_pmulti_dset_ath_run [HPCS-8110] source/reproducers/mpi/hdf5_paralle_incorrect:hdf5_paralle_incorrect_testphdf5_ath_run [HPCS-8110] source/reproducers/openmp/findclus:findclus_ath_run [GSD-5335] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20 [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/gamess_mini_all_kernels:gamess_mini_all_kernels_j20_reduction [CMPLRLLVM-45082,CMPLRLLVM-45890,CMPLRTST-21020,IGC-6246,XDEPS-5609] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/map_to_many_sync:map_to_many_sync_simple_ath_run [CMPLRLLVM-32920] source/reproducers/openmp/reduction_complex_perf:reduction_complex_perf_ath_run [CMPLRLLVM-37471,CMPLRLLVM-38249,CMPLRLLVM-42440] P0 ``` # Regressions for oneapi/eng-compiler/2023.05.15.007 compared to oneapi/eng-compiler/2023.05.15.006 ``` source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_opcode source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_bad_operand_syntax source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_duplicate_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_illegal_exec_size source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_label source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_missing_region source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_simple source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_decl source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_undefined_pred source/reproducers/dpcpp/ms371-InlineAsm:ms371-InlineAsm_asm_wrong_declare source/reproducers/dpcpp/multigpu_multiqueue source/reproducers/dpcpp/syclmemcpy2d:syclmemcpy2d_ath_run [GSD-5515] source/reproducers/dpcpp/vecadd_v0_multidevice source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 [CMPLRLLVM-39495] source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 [CMPLRLLVM-39495] source/reproducers/mpi/openmp_target_zam:openmp_target_zam_zam1_ath_run [HPCS-8169] ``` ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <[email protected]> Sent: Tuesday, July 25, 2023 4:58 PM To: [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions An updated version of the Aurora SDK is now available and will be made default Wednesday at 12pm CDT. The module files for this release are: oneapi/eng-compiler/2023.05.15.006 – This will be made default, it is the 2023.05.15.003 release with the UMD updated to 647 oneapi/eng-compiler/2023.05.15.007 – Updated engineering compiler/mkl/mpich + UMD 647 This is based on the 2023.1 release of the oneAPI SDK. Colleen will provide caveats, workarounds and other useful information about these releases shortly. -- Christopher Chan-Nui [email protected] From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <[email protected]> Date: Thursday, June 22, 2023 at 1:26 PM To: [email protected] <[email protected]> Subject: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions A new version of the Aurora SDK is now available. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release are: oneapi/release/2023.05.15.001 oneapi/eng-compiler/2023.05.15.003 This is based on the 2023.1 release of the oneAPI SDK. Release notes are available at https://www.intel.com/content/www/us/en/developer/articles/release-notes/int... https://www.intel.com/content/www/us/en/developer/articles/release-notes/int... Addendums for the engineering components are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. -- Christopher Chan-Nui [email protected]