May 2022 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Re: [Ecp-aurora-sdk-discuss] Configuring Spack package for Intel GPUs
by Garcia Martinez, Marta 31 May '22

31 May '22

Hi Erik, You have some information about E4S Spack packages installations on facilities (included JLSE) here: https://e4s.readthedocs.io/en/latest/facility_e4s.html In this file from Frank Willmore when he was at ANL you can see AMReX but it’s with gcc https://github.com/spack/spack-configs/blob/main/ANL/JLSE/Arcticus/E4S-21.1… I know that it’s not the same package but for Kokkos Kernels, we are using this (as an example) - kokkos(a)3.6.00 %oneapi(a)2022.1.0 +sycl +openmp std=17 +tests +examples - kokkos@develop %oneapi(a)2022.1.0 +sycl +openmp std=17 +tests +examples - kokkos-kernels(a)3.6.00 %oneapi(a)2022.1.0 +openmp build_type=Release ^[email protected]%[email protected]+sycl+openmp +tests +examples - kokkos-kernels@develop %oneapi(a)2022.1.0 +openmp build_type=Release ^kokkos@develop%[email protected]+sycl+openmp +tests +examples With this (since the default is pointing to @[email protected]) - compiler: spec: oneapi(a)2022.1.0 paths: cc: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/compiler/pseudo-20220223/compiler/linux/bin/icx cxx: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/compiler/pseudo-20220223/compiler/linux/bin/icpx f77: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/compiler/pseudo-20220223/compiler/linux/bin/ifx fc: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/compiler/pseudo-20220223/compiler/linux/bin/ifx Hope it helps, Marta On May 23, 2022, at 10:56 AM, Harms, Kevin via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: Erik, when you say "I compile directly from the command line", i assume you mean that you are loading the modules that JLSE provides? When you use spack, it clears all of the environment, so it's possible you might need to modify spack to find our compiler install with 'find compilers'. Another option, perhaps unsafe, is to use the '--dirty' option on spack which won't clear the environment. kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Erik Palmer via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Monday, May 23, 2022 10:09 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [Ecp-aurora-sdk-discuss] Configuring Spack package for Intel GPUs Hi All, I am trying to configure the AMReX package for Spack to run with an intel gpu backend and I am running into some issues. I realize this question may overlap with a few areas (like Spack), but any advice you can offer would be appreciated. My issue is that when I try to install AMReX via spack with the command `spack install amrex+scyl%oneapi@2023` I get an error message at the CMake build step that says: Could NOT find IntelDPCPP (missing: SYCL_INCLUDE_DIR SYCL_LIBRARY_DIR) However, when I compile directly from the command line, CMake seems to find the package without issue and compiles the project. Any ideas on what I could be missing? Thanks! Erik Palmer -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] Non-uniform work-groups not supported?
by Vargas, Arturo 26 May '22

26 May '22

Thanks! Not important, I misunderstood how to setup the compute grid. Just learned that when setting up a the nd_range (cl::sycl::nd_range<3>(gridSize, blockSize). The grid size is the total number of work-items on the grid. -Arturo From: "Applencourt, Thomas" <tapplencourt(a)anl.gov> Date: Thursday, May 26, 2022 at 9:14 AM To: "Vargas, Arturo" <vargas45(a)llnl.gov> Cc: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: Non-uniform work-groups not supported? Got the confirmation from Intel: > Correct – it is not currently supported. Folks are looking into it, but it’s not likely to appear this year. If it's important for your project, we can prioritize it. Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Applencourt, Thomas <tapplencourt(a)anl.gov> Sent: Thursday, May 26, 2022 09:50 To: Vargas, Arturo <vargas45(a)llnl.gov> Subject: Re: Non-uniform work-groups not supported? Hi, I think it will indeed not be supported. Indeed the L0 (our runtime backend), as NVIDIA, is "block" and "thread-per-block". That makes supporting non-unfirom work-groups tricky. But maybe something can be implemented in the runtime. Let me double-check and I will get back to you, Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Vargas, Arturo via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, May 26, 2022 09:37 To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Non-uniform work-groups not supported? Hi all, I’m trying to run on arcticus with non-uniform workgroups and it looks like its not supported? I hit the following error: Terminate called after throwing an instance of 'cl::sycl::nd_range_error' what(): Non-uniform work-groups are not supported by the target device -54 (CL_INVALID_WORK_GROUP_SIZE) Aborted Is this expected and will be supported in future hardware? Best, Arturo

1 0

Re: [Ecp-aurora-sdk-discuss] Non-uniform work-groups not supported?
by Applencourt, Thomas 26 May '22

26 May '22

Got the confirmation from Intel: > Correct – it is not currently supported. Folks are looking into it, but it’s not likely to appear this year. If it's important for your project, we can prioritize it. Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Applencourt, Thomas <tapplencourt(a)anl.gov> Sent: Thursday, May 26, 2022 09:50 To: Vargas, Arturo <vargas45(a)llnl.gov> Subject: Re: Non-uniform work-groups not supported? Hi, I think it will indeed not be supported. Indeed the L0 (our runtime backend), as NVIDIA, is "block" and "thread-per-block". That makes supporting non-unfirom work-groups tricky. But maybe something can be implemented in the runtime. Let me double-check and I will get back to you, Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Vargas, Arturo via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, May 26, 2022 09:37 To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Non-uniform work-groups not supported? Hi all, I’m trying to run on arcticus with non-uniform workgroups and it looks like its not supported? I hit the following error: Terminate called after throwing an instance of 'cl::sycl::nd_range_error' what(): Non-uniform work-groups are not supported by the target device -54 (CL_INVALID_WORK_GROUP_SIZE) Aborted Is this expected and will be supported in future hardware? Best, Arturo

1 0

Non-uniform work-groups not supported?
by Vargas, Arturo 26 May '22

26 May '22

Hi all, I’m trying to run on arcticus with non-uniform workgroups and it looks like its not supported? I hit the following error: Terminate called after throwing an instance of 'cl::sycl::nd_range_error' what(): Non-uniform work-groups are not supported by the target device -54 (CL_INVALID_WORK_GROUP_SIZE) Aborted Is this expected and will be supported in future hardware? Best, Arturo

1 0

Re: [Ecp-aurora-sdk-discuss] Incorrect env variable causing "cl::sycl::runtime_error" with intel OneAPI?
by Muralidharan, Servesh 25 May '22

25 May '22

Hi Erik, Please take a look at the intel_compute_runtime module that is loaded along with oneapi module and I suspect you need to manually load it inside the spack env. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Erik Palmer via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Wednesday, May 25, 2022 at 1:21 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Incorrect env variable causing "cl::sycl::runtime_error" with intel OneAPI? Hi All, Thanks to previous suggestions from this mailing list, I was able to install amrex with an intel gpu backend using spack. Now I am trying to enable the built-in smoke test with spack for amrex and am getting a runtime error when I call `spack test run amrex` (for the intel gpu enabled version of amrex). The error is 1 error found in test log: 48 [100%] Built target install_test 49 ==> [2022-05-25-16:43:26.093459] AMReX Stand-Alone Smoke Test -- AmrCore 50 ==> [2022-05-25-16:43:26.093895] './install_test' './cache/amrex/Tests/Amr/Advection_AmrCore/Exec/inputs-ci' 51 Initializing oneAPI... 52 terminate called after throwing an instance of 'cl::sycl::runtime_error' 53 what(): No device of requested type available. Please check https://software.intel.com/content/www/us/en/develop/articles/intel-oneapi-… ml -1 (CL_DEVICE_NOT_FOUND) >> 54 FAILED: Command exited with status -6: When I call `spack test run --dirty amrex`, it passes without issue. This leads me to believe an environment variable is causing the runtime error. Has anyone come across this error before and possibly knows which one is missing/incorrect to cause this runtime error? Thanks for any advice! Erik Palmer

1 0

Incorrect env variable causing "cl::sycl::runtime_error" with intel OneAPI?
by Erik Palmer 25 May '22

25 May '22

Hi All, Thanks to previous suggestions from this mailing list, I was able to install amrex with an intel gpu backend using spack. Now I am trying to enable the built-in smoke test with spack for amrex and am getting a runtime error when I call `spack test run amrex` (for the intel gpu enabled version of amrex). The error is 1 error found in test log: 48 [100%] Built target install_test 49 ==> [2022-05-25-16:43:26.093459] AMReX Stand-Alone Smoke Test -- AmrCore 50 ==> [2022-05-25-16:43:26.093895] './install_test' './cache/amrex/Tests/Amr/Advection_AmrCore/Exec/inputs-ci' 51 Initializing oneAPI... 52 terminate called after throwing an instance of 'cl::sycl::runtime_error' 53 what(): No device of requested type available. Please check https://software.intel.com/content/www/us/en/develop/articles/intel-oneapi-… ml -1 (CL_DEVICE_NOT_FOUND) >> 54 FAILED: Command exited with status -6: When I call `spack test run --dirty amrex`, it passes without issue. This leads me to believe an environment variable is causing the runtime error. Has anyone come across this error before and possibly knows which one is missing/incorrect to cause this runtime error? Thanks for any advice! Erik Palmer

1 0

Aurora Applications Biweekly meeting
by Knight, Christopher J. 25 May '22

25 May '22

Hi everyone on the ecp-aurora-sdk-discuss mailing list, Over the past few years, we at Argonne have been holding a biweekly Aurora Applications meeting where Argonne and Intel points-of-contacts discuss recent progress by teams and any challenges faced on the Intel GPU testbed at JLSE in preparation for Aurora. The focus is very much on the use of Intel GPUs and oneAPI software stack, but comparisons with other hardware to explore performance portability topics is a common theme. Up until recently, this meeting had been restricted to only Argonne and Intel personnel on the appropriate NDA. With PVC nodes now available in JLSE, we have the opportunity to open up this meeting to the broader developer community within the ESP and ECP projects. Attendees are still required to be on the appropriate NDA, but this now includes everyone on this ecp-aurora-sdk-discuss list with JLSE access to the Intel GPU testbed. This is a biweekly meeting on Fridays at 1pm CT with typically two or more planned presentations and Q&A discussion. We try to plan meeting agendas out several months, but there is flexibility in the schedule and we welcome suggestions/volunteers for upcoming presentations and discussions. The next meeting is Friday, June 10th. For those that are interested in participating, please send Chris Knight (knightc(a)anl.gov<mailto:[email protected]>) or Scott Parker (sparker(a)anl.gov) an email (or slack message) in advance of the scheduled meeting. This "pre-registration" is required as we need to verify the NDA coverage for people in these meetings and only those that have been pre-approved will be able to participate. Thank you, and let us know if you have any questions. Chris & Scott (+Tim & Venkat)

1 0

Re: [Ecp-aurora-sdk-discuss] Configuring Spack package for Intel GPUs
by Harms, Kevin 23 May '22

23 May '22

Erik, when you say "I compile directly from the command line", i assume you mean that you are loading the modules that JLSE provides? When you use spack, it clears all of the environment, so it's possible you might need to modify spack to find our compiler install with 'find compilers'. Another option, perhaps unsafe, is to use the '--dirty' option on spack which won't clear the environment. kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Erik Palmer via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Monday, May 23, 2022 10:09 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov Subject: [Ecp-aurora-sdk-discuss] Configuring Spack package for Intel GPUs Hi All, I am trying to configure the AMReX package for Spack to run with an intel gpu backend and I am running into some issues. I realize this question may overlap with a few areas (like Spack), but any advice you can offer would be appreciated. My issue is that when I try to install AMReX via spack with the command `spack install amrex+scyl%oneapi@2023` I get an error message at the CMake build step that says: Could NOT find IntelDPCPP (missing: SYCL_INCLUDE_DIR SYCL_LIBRARY_DIR) However, when I compile directly from the command line, CMake seems to find the package without issue and compiles the project. Any ideas on what I could be missing? Thanks! Erik Palmer

1 0

Configuring Spack package for Intel GPUs
by Erik Palmer 23 May '22

23 May '22

Hi All, I am trying to configure the AMReX package for Spack to run with an intel gpu backend and I am running into some issues. I realize this question may overlap with a few areas (like Spack), but any advice you can offer would be appreciated. My issue is that when I try to install AMReX via spack with the command `spack install amrex+scyl%oneapi@2023` I get an error message at the CMake build step that says: Could NOT find IntelDPCPP (missing: SYCL_INCLUDE_DIR SYCL_LIBRARY_DIR) However, when I compile directly from the command line, CMake seems to find the package without issue and compiles the project. Any ideas on what I could be missing? Thanks! Erik Palmer

1 0

Re: [Ecp-aurora-sdk-discuss] New Engineering Compiler and UMD update
by Bertoni, Colleen 17 May '22

17 May '22

Thanks Chris! Like Chris said, with 007 fixed most of the application issues we saw in 006. In particular, the issues in 006 with Thermo4PFM, WEST, QE, nekRS, and miniQMC are now fixed. AMR-Wind, GAMESS, and KokkosUnitTests are still failing. Compared to the 006 SDK, we had 32 fixes and 6 regressions for the bug reproducers. The overall results for reproducers is below. Thanks a lot for reporting bugs! # Fixes for 007 SDK vs. 006 SDK source/reproducers/dpcpp/sycl2020_reduction_ats_implicit [XDEPS-3555] source/reproducers/dpcpp/sycl_enable_profiling [CMPLRLLVM-35319] source/reproducers/hybrid/concurency:concurency_omp_host_threads_C_H2D_ath_run source/reproducers/hybrid/concurency:concurency_omp_host_threads_C_M2D_ath_run source/reproducers/hybrid/concurency:concurency_sycl_con_in_order_C_H2D_ath_run source/reproducers/hybrid/concurency:concurency_sycl_con_in_order_C_M2D_ath_run source/reproducers/hybrid/concurency:concurency_sycl_con_out_of_order_C_H2D_ath_run source/reproducers/hybrid/concurency:concurency_sycl_con_out_of_order_C_M2D_ath_run source/reproducers/hybrid/reduction:reduction_sycl [CMPLRLLVM-30136] source/reproducers/ifx/array_copy:array_copy_array_copy_gfortran_ath_run [CMPLRLLVM-35977] source/reproducers/ifx/array_copy:array_copy_array_copy_gfortran_debug_ath_run [CMPLRLLVM-35977] source/reproducers/ifx/array_copy:array_copy_array_copy_ifx_ath_run [CMPLRLLVM-35977] source/reproducers/ifx/gamess_compile_fail source/reproducers/ifx/qe_fftxlib_test_crash:qe_fftxlib_test_crash_fftxlib_test [CMPLRLLVM-24637] source/reproducers/l0/append_wait_on_events source/reproducers/mkl/thread_safe_check [MKLD-12734] P0 source/reproducers/opencl/device_timer [XDEPS-1275] source/reproducers/openmp/global_math:global_math_ath_compile [CMPLRLLVM-31135] source/reproducers/openmp/global_math:global_math_ath_run [CMPLRLLVM-31135] source/reproducers/openmp/icx_fast_spirv [CMPLRLLVM-36657] source/reproducers/openmp/ifx_customreduction_collapse_ice:ifx_customreduction_collapse_ice_ice [CMPLRLLVM-28708] P0 source/reproducers/openmp/ifx_customreduction_collapse_ice:ifx_customreduction_collapse_ice_workaround_wrong_value [CMPLRLLVM-28708] P0 source/reproducers/openmp/map_pinned_host_mem source/reproducers/openmp/map_pinned_host_mem:map_pinned_host_mem_simple source/reproducers/openmp/miniqmc-debug [CMPLRLLVM-34830] P0 source/reproducers/openmp/miniqmc-smallrun [CMPLRLLVM-27865] P0 source/reproducers/openmp/omp_fortran_struct_pointer_target_map:omp_fortran_struct_pointer_target_map_ath_run [CMPLRLLVM-28670] source/reproducers/openmp/one_reduction:one_reduction_fortran_custom_ath_run [CMPLRLLVM-22031,XDEPS-3875] P0 source/reproducers/openmp/proc_bind source/reproducers/openmp/static_global_funct:static_global_funct_ath_compile [CMPLRLLVM-28570] source/reproducers/openmp/static_global_funct:static_global_funct_ath_run [CMPLRLLVM-28570] source/reproducers/openmp/target_enter_exit:target_enter_exit_enter_exit # Regressions for 007 SDK vs. 006 SDK source/reproducers/dpcpp/hang_atomic_on_local:hang_atomic_on_local-native_affmask_1_128_ath_run source/reproducers/hybrid/concurency:concurency_omp_host_threads_C_C_WA_ath_run source/reproducers/ifx/array_copy:array_copy_array_copy_ifort_ath_run [CMPLRLLVM-35977] source/reproducers/ifx/array_copy:array_copy_array_copy_ifort_debug_ath_run [CMPLRLLVM-35977] source/reproducers/l0/hang_barrier_profiling:hang_barrier_profiling_ath_run source/reproducers/l0/hang_barrier_profiling:hang_barrier_profiling_wa_ath_run ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Monday, May 16, 2022 9:26 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] New Engineering Compiler and UMD update A new version of the compiler and update UMD (438) are now available (but not default) on the JLSE cluster. oneapi/eng-compiler/2022.01.30.007 This addresses issues found in the 2022.01.30.006 release. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]>

1 0