September 2022 - Ecp-aurora-sdk-discuss

Re: [Ecp-aurora-sdk-discuss] Florentia node upgrade
by Muralidharan, Servesh 21 Sep '22

21 Sep '22

Hello Florentia node users, The upgrade process for one of the PVC nodes is complete and we would be releasing it to users today at 6:00PM CST. Once the other node’s upgrade is completed, it would be added to the same queues as well. There are many jobs queued up on florentia/florentia_debug queue which will start running on release, however depending on several changes detailed below your jobs might be affected. The SDK environment would be the same as you would use in Arcticus, but please take note of the changes below, 1. The default env settings would match what’s in Arcticus today i.e. multi_gpu+implicit_scaling 2. The affinity mask setting to default to the first tile of the 4 GPUs will be removed. 3. To make it easier for users to deal with GPU env options, we will introduce several meta modulefiles one can load to configure the env. This is something that is specific to JLSE and not an Intel recommended solution. So, you are welcome to override with specific env option, should you choose. But for any complaints/feedback please reach out to me. * To take a look at available options: `module av intel_gpu_env` * To do implicit scaling: `module load intel_gpu_env/implicit_scaling` * To do explicit scaling: `module load intel_gpu_env/explicit_scaling` * If you want the defaults for the platform: `module load intel_gpu_env/platform_default` * There are many others that we are testing for their usefulness. There will be a presentation on details about this in the next apps presentation. Please tune it to hear more details. There are couple of other SDK/runtime changes that one needs to be mindful off as well, OneAPI module Version SDK Version UMD Arcticus Node GPU AOT String Florentia Node GPU AOT String oneapi/eng-compiler/2022.06.30.002 (default) 2022.2 Public + Engineering Components agama-prerelease-475 12.1.0 12.4.1 oneapi/eng-compiler/2022.06.30.004 2022.2 Public + Engineering Components agama-devel-449 12.50.4 12.60.7 oneapi/eng-compiler/2022.10.15.002 2022.3 Public + Engineering Components agama-devel-449 12.50.4 12.60.7 oneapi/release/2022.06.30.001 2022.2 Public agama-prerelease-475 12.1.0 12.4.1 oneapi/release/2022.06.30.003 2022.2 Public agama-devel-449 12.50.4 12.60.7 oneapi/release/2022.10.15.001 2022.3 Public agama-devel-449 12.50.4 12.60.7 Example: 2022.06.30.002 and 2022.06.30.004 are identical, except the latter loads a newer runtime that we have found helps in some apps. Please ensure you load the corresponding Kokkos modulefiles when you do so. Neverthless, recommendation is to try the latest SDK drop and see if that gives the best functionality/performance for your application, although there could be some bleeding edge changes that might affect your apps. We will have a presentation scheduled on the next Apps WG call, with the details on the hardware upgrades. Please reach out to me for any feedback/comments. On behalf of early hardware team. -- Servesh

1 0

Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions
by Bertoni, Colleen 20 Sep '22

20 Sep '22

Thanks Chris! First some important things to note about this drop: 1) If you used AOT compiling: The names for the devices has changed in the newer runtime. The changes are: xehp --> 12.50.4 12.1.0 --> 12.50.4 And for the PVC B4, use 12.60.7 2) The SYCL include tree has changed. <Cl/sycl.hpp> is deprecated, please use <sycl/sycl.hpp> instead. Also, for "#include <CL/sycl/backend/level_zero.hpp>" replace with "#include <sycl/ext/oneapi/backend/level_zero.hpp>" Overall results: Compared to the 2022.06.30.002 SDK, we had 20 fixes and 20 regressions for the bug reproducers on ATS (however, 4 of the regressions are just due to the changes listed above). On PVC B4, there were 35 fixes and 19 regressions (with 4 of the regressions due to the changes listed above). The overall results for reproducers is below. For the regressions, Intel has opened new JIRAs for them if they are still failing on newer internal compilers (thanks Brian!) Thanks a lot to everyone for reporting bugs! For applications, the previous heFFTe ICE is fixed, and the osiris is now compiling. Many of the applications in the test set are failing because of the two changes above. If you're the application owner, we'll reach out and let you in the next day. Let us know if you have any questions, and thanks for the bug reports! Thanks, Colleen # Fixes for 2022.10.15.002 SDK on ATS vs. 2022.06.30.002 SDK on ATS source/reproducers/dpcpp/vector_nontemporal [CMPLRLLVM-39266] source/reproducers/dpcpp_ct/dpct_constant_iterator source/reproducers/general/config_flag_check:config_flag_check_dpcpp [CMPLRLLVM-38867] source/reproducers/general/config_flag_check:config_flag_check_ifx [CMPLRLLVM-38867] source/reproducers/icx/dbl_true_min source/reproducers/ifx/gene_miller_geom_segfault:gene_miller_geom_segfault_compile [CMPLRLLVM-39118] P0 source/reproducers/mkl/usm_syr2k [MKLD-13737] P0 source/reproducers/mpi/aot_compile source/reproducers/mpi/implicit_scaling:implicit_scaling_mpi_vs_omp source/reproducers/openmp/aot_example:aot_example_ifx [CMPLRLLVM-38868] source/reproducers/openmp/array_red_fortran [CMPLRLLVM-36508] source/reproducers/openmp/fortran_asm_link:fortran_asm_link_simple source/reproducers/openmp/long_round:long_round_ath_run [CMPLRLLVM-37210] P0 source/reproducers/openmp/ms371-InlineAsmOmp:ms371-InlineAsmOmp_atomic_update_add [XDEPS-4366] source/reproducers/openmp/ms371-InlineAsmOmp:ms371-InlineAsmOmp_critical_ze_binary [XDEPS-4366] source/reproducers/openmp/ms371-InlineAsmOmp:ms371-InlineAsmOmp_target_teams_distribute [XDEPS-4366] source/reproducers/openmp/omp_visa:omp_visa_target_teams_distribute [CMPLRXDEPS-705,XDEPS-2276] P0 source/reproducers/openmp/osiris_ice [CMPLRLLVM-39138] source/reproducers/openmp/simple_local_allocate:simple_local_allocate_omp_alloc source/reproducers/openmp/template_stack [CMPLRLLVM-37965] P0 # Regressions for 2022.10.15.002 SDK on ATS vs. 2022.06.30.002 SDK on ATS source/reproducers/dpcpp/ATSsubdevice_regression source/reproducers/dpcpp/NWChemEx-CCSDT source/reproducers/dpcpp/sycl_L0_peer2peer:sycl_L0_peer2peer_P2P_full [CMPLRLLVM-34952] P0 source/reproducers/dpcpp/sycl_L0_peer2peer:sycl_L0_peer2peer_P2P_tile [CMPLRLLVM-34952] P0 source/reproducers/hybrid/interop source/reproducers/ifx/phasta_target_simd [CMPLRLLVM-35621] P0 source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 source/reproducers/mkl/sparse_openmp_kokkos_kernels:sparse_openmp_kokkos_kernels_europe_osm_ath_run source/reproducers/mkl/thread_safe_check:thread_safe_check_ze_binary_ath_run [MKLD-12734,XDEPS-4369,XDEPS-4403] P0 source/reproducers/mpi/implicit_scaling:implicit_scaling_implicit_vs_explicit source/reproducers/openmp/fortran_nested_function_calls [CMPLRLLVM-27380] source/reproducers/openmp/l0_follow_the_spec P0 source/reproducers/openmp/map_pinned_host_mem source/reproducers/openmp/qmcpack.cpu.qmcpack.ompsimd.sincos [CMPLRLLVM-24010] source/reproducers/openmp/short-vector-math:short-vector-math.r.O3.fastmath.fiopenmp [CMPLRLLVM-11184,CMPLRLLVM-18505,CMPLRLLVM-20040,CMPLRLLVM-21077] P0 source/reproducers/openmp/short-vector-math:short-vector-math.r.fiopenmp [CMPLRLLVM-11184,CMPLRLLVM-18505,CMPLRLLVM-20040,CMPLRLLVM-21077] P0 source/reproducers/openmp/slow_reduction:slow_reduction_aot [CMPLRLLVM-36250] source/reproducers/openmp/slow_zeCommandQueueExecuteCommandLists:slow_zeCommandQueueExecuteCommandLists_ath_run [CMPLRLLVM-32940,XDEPS-3448,XDEPS-3829] P0 source/reproducers/openmp/slow_zeCommandQueueExecuteCommandLists:slow_zeCommandQueueExecuteCommandLists_hanging_ath_run [CMPLRLLVM-32940,XDEPS-3448,XDEPS-3829] P0 # Fixes for 2022.10.15.002 SDK on PVC vs. 2022.06.30.002 SDK on ATS source/reproducers/dpcpp/vector_nontemporal [CMPLRLLVM-39266] source/reproducers/dpcpp_ct/dpct_constant_iterator source/reproducers/general/config_flag_check:config_flag_check_dpcpp [CMPLRLLVM-38867] source/reproducers/general/config_flag_check:config_flag_check_ifx [CMPLRLLVM-38867] source/reproducers/hybrid/oneConcurency:oneConcurency_omp_host_threads_H2D_D2H_ath_run [CMPLRLLVM-34779,XDEPS-3493,XDEPS-2202,CMPLRLLVM-34779,XDEPS-3493,XDEPS-2202] P0 source/reproducers/hybrid/oneConcurency:oneConcurency_omp_host_threads_M2D_D2M_ath_run [CMPLRLLVM-34779,XDEPS-3493,XDEPS-2202,CMPLRLLVM-34779,XDEPS-3493,XDEPS-2202] P0 source/reproducers/icx/dbl_true_min source/reproducers/ifx/gene_miller_geom_segfault:gene_miller_geom_segfault_compile [CMPLRLLVM-39118] P0 source/reproducers/igc/quda_segfault:quda_segfault_pvc_older_runtime source/reproducers/l0/concurrent_please:concurrent_please_immediate_ath_run source/reproducers/l0/concurrent_please:concurrent_please_list_batched_ath_run source/reproducers/l0/hang_barrier_profiling:hang_barrier_profiling_ath_run source/reproducers/l0/hang_barrier_profiling:hang_barrier_profiling_wa_ath_run source/reproducers/mkl/usm_syr2k [MKLD-13737] P0 source/reproducers/mpi/aot_compile source/reproducers/openmp/aot_example:aot_example_ifx [CMPLRLLVM-38868] source/reproducers/openmp/array_red_fortran [CMPLRLLVM-36508] source/reproducers/openmp/atomic_hanging [CMPLRLLVM-35605,XDEPS-3798] source/reproducers/openmp/fortran_asm_link:fortran_asm_link_simple source/reproducers/openmp/hang_glinetable_plus_large_register_file:hang_glinetable_plus_large_register_file_ath_run [CMPLRLLVM-36221] source/reproducers/openmp/large_transfer_incorect:large_transfer_incorect_ath_run [XDEPS-2757] P0 source/reproducers/openmp/long_round:long_round_ath_run [CMPLRLLVM-37210] P0 source/reproducers/openmp/memory_fail [CMPLRLLVM-37979] P0 source/reproducers/openmp/modf_team:modf_team_ath_run [CMPLRLLVM-37476] P0 source/reproducers/openmp/modf_team:modf_team_subdevice_ath_run [CMPLRLLVM-37476] P0 source/reproducers/openmp/ms371-InlineAsmOmp:ms371-InlineAsmOmp_atomic_update_add [XDEPS-4366] source/reproducers/openmp/ms371-InlineAsmOmp:ms371-InlineAsmOmp_critical_ze_binary [XDEPS-4366] source/reproducers/openmp/ms371-InlineAsmOmp:ms371-InlineAsmOmp_target_teams_distribute [XDEPS-4366] source/reproducers/openmp/omp_present_table_performance [CMPLRLLVM-28738] P0 source/reproducers/openmp/omp_visa:omp_visa_target_teams_distribute [CMPLRXDEPS-705,XDEPS-2276] P0 source/reproducers/openmp/osiris_ice [CMPLRLLVM-39138] source/reproducers/openmp/simple_local_allocate:simple_local_allocate_omp_alloc source/reproducers/openmp/template_stack [CMPLRLLVM-37965] P0 source/reproducers/openmp/template_stack:template_stack_firstprivate [CMPLRLLVM-37965] P0 source/reproducers/openmp/w60_gamess_rimp2 [XDEPS-2757] # Regressions for 2022.10.15.002 SDK on PVC vs. 2022.06.30.002 SDK on ATS source/reproducers/dpcpp/ATSsubdevice_regression source/reproducers/dpcpp/NWChemEx-CCSDT source/reproducers/dpcpp/sycl_L0_peer2peer:sycl_L0_peer2peer_P2P_full [CMPLRLLVM-34952] P0 source/reproducers/dpcpp/sycl_L0_peer2peer:sycl_L0_peer2peer_P2P_tile [CMPLRLLVM-34952] P0 source/reproducers/hybrid/interop source/reproducers/ifx/phasta_target_simd [CMPLRLLVM-35621] P0 source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_10 source/reproducers/mkl/SYCL_USM_MKL_L0_subdevice:SYCL_USM_MKL_L0_subdevice_11 source/reproducers/mkl/getrsnp_batch_runtime [MKLD-12958] source/reproducers/mkl/sparse_openmp_kokkos_kernels:sparse_openmp_kokkos_kernels_amazon0302_ath_run source/reproducers/mkl/sparse_openmp_kokkos_kernels:sparse_openmp_kokkos_kernels_europe_osm_ath_run source/reproducers/openmp/fortran_nested_function_calls [CMPLRLLVM-27380] source/reproducers/openmp/fortran_simd_issue:fortran_simd_issue_simdlen [CMPLRLLVM-26465] source/reproducers/openmp/gamess_mini_jit [XDEPS-2759] source/reproducers/openmp/map_pinned_host_mem source/reproducers/openmp/memcopy_multidevice source/reproducers/openmp/qmcpack.cpu.qmcpack.ompsimd.sincos [CMPLRLLVM-24010] source/reproducers/openmp/short-vector-math:short-vector-math.r.O3.fastmath.fiopenmp [CMPLRLLVM-11184,CMPLRLLVM-18505,CMPLRLLVM-20040,CMPLRLLVM-21077] P0 source/reproducers/openmp/short-vector-math:short-vector-math.r.fiopenmp [CMPLRLLVM-11184,CMPLRLLVM-18505,CMPLRLLVM-20040,CMPLRLLVM-21077] P0 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, September 20, 2022 5:06 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions The new modulefiles are now available. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Chan-nui, Christopher <christopher.chan-nui(a)intel.com> Date: Tuesday, September 20, 2022 at 12:06 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: New oneAPI release and eng-compiler versions At 5pm CDT the new version of the Aurora SDK will be released. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release will be: oneapi/release/2022.10.15.001 oneapi/eng-compiler/2022.10.15.002 Release Notes for 2022.3 PUBLIC<https://wiki.jlse.anl.gov/download/attachments/26214592/SDK-Release-Notes_2…> are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. This update will move the Base/HPC install to 2022.3 Public (mostly security fixes). It also updates mpich to drop 49.1. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] New oneAPI release and eng-compiler versions
by Chan-nui, Christopher 20 Sep '22

20 Sep '22

The new modulefiles are now available. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Chan-nui, Christopher <christopher.chan-nui(a)intel.com> Date: Tuesday, September 20, 2022 at 12:06 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: New oneAPI release and eng-compiler versions At 5pm CDT the new version of the Aurora SDK will be released. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release will be: oneapi/release/2022.10.15.001 oneapi/eng-compiler/2022.10.15.002 Release Notes for 2022.3 PUBLIC<https://wiki.jlse.anl.gov/download/attachments/26214592/SDK-Release-Notes_2…> are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. This update will move the Base/HPC install to 2022.3 Public (mostly security fixes). It also updates mpich to drop 49.1. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

New oneAPI release and eng-compiler versions
by Chan-nui, Christopher 20 Sep '22

20 Sep '22

At 5pm CDT the new version of the Aurora SDK will be released. This will NOT be enabled as the default version. Running jobs will NOT be affected. The module files for this release will be: oneapi/release/2022.10.15.001 oneapi/eng-compiler/2022.10.15.002 Release Notes for 2022.3 PUBLIC<https://wiki.jlse.anl.gov/download/attachments/26214592/SDK-Release-Notes_2…> are available at https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Status+and+Notes. This update will move the Base/HPC install to 2022.3 Public (mostly security fixes). It also updates mpich to drop 49.1. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Florentia node upgrade
by Muralidharan, Servesh 19 Sep '22

19 Sep '22

Hello Florentia node users, We are planning hardware upgrades to the Florentia nodes this week. So, expect node availability to be limited until we complete this process. Will send another announce once the upgrade process is completed. -- Servesh

1 0

Question about distributed FFT usage on GPUs
by Bertoni, Colleen 12 Sep '22

12 Sep '22

Hi everyone, We have a quick question from the MKL team. Are you or any application you're working with using the Nvidia distributed FFT APIs? That is, either - The cuFFTMp APIs https://developer.nvidia.com/blog/multinode-multi-gpu-using-nvidia-cufftmp-… - Or the cufftXt APIs https://docs.nvidia.com/cuda/cufft/index.html#multiple-GPU Thanks, Colleen

1 0