April 2022 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Re: [Ecp-aurora-sdk-discuss] Upcoming Engineering Compiler and UMD updates
by Bertoni, Colleen 21 Apr '22

21 Apr '22

Thanks Chris! With the new oneapi/eng-compiler/2022.01.30.006 (non-default) drop, we had 13 regressions and 33 fixes. In particular, reduction issues that we had in the last SDK were fixed. But there were some compile-time issues with full applications, so we decided not to make this SDK default. See below for details. Thanks a lot to the people who added applications and bug reproducers to the test set, it enabled us to track the issues and report them as soon as possible to Intel. Hopefully this also makes using the SDK on JLSE a nicer experience! We also started tracking performance for the tests which have FOMs in them.If you want to add an FOM to a test in the test set see here: https://wiki.jlse.anl.gov/display/inteldga/Testing+setup+and+examples Thanks, Colleen # Applications issues: ## Regressions: AMR-Wind (CMPLRLLVM-37056, CMPLRLLVM-37044, a workaround is to use lower optimization level and remove -fsycl-device-code-split=per_kernel) Thermo4PFM (CMPLRLLVM-36839) WEST (CMPLRLLVM-36886, a workaround is to use a lower optimization level) QE (CMPLRLLVM-36886, a workaround is to use a lower optimization level) GAMESS (compile fail with `-g`, Fixed with 20220419 (22.0-1524) compiler ) KokkosUnitTests (8 unit tests are failing) miniqmc-smallrun (CMPLRLLVM-36880, link time fail) ## Fixes: All the OpenMC tests (with workarounds) we have in the test set are now all passing # Bug Reproducers: ## Fixes: source/reproducers/dpcpp/invalidArraySize [CMPLRLLVM-31853] source/reproducers/dpcpp/std_complex_double [CORC-7513] source/reproducers/dpcpp/submit_long [CMPLRLLVM-32601] P0 source/reproducers/dpcpp/sycl_enable_profiling [CMPLRLLVM-35319] source/reproducers/hybrid/omp_dpcpp_memory_interop:omp_dpcpp_memory_interop_MS67 [CMPLRXDEPS-856,XDEPS-3874] source/reproducers/icx/sycl_icx [CMPLRLLVM-33525] source/reproducers/ifx/array_copy:array_copy_array_copy_ifx_ath_run [CMPLRLLVM-35977] source/reproducers/ifx/array_copy:array_copy_array_copy_ifx_debug_ath_run [CMPLRLLVM-35977] source/reproducers/ifx/associate_ptr:associate_ptr_ath_run [CMPLRLLVM-30867] source/reproducers/igc/cpp_jit_fail [XDEPS-3390] source/reproducers/igc/e3sm_warnings [XDEPS-3389] source/reproducers/igc/fortran_omp_segfault [XDEPS-3206] source/reproducers/igc/warnings_xgc_tim source/reproducers/l0/hang_barrier_profiling:hang_barrier_profiling_ath_run source/reproducers/mkl/c2c_1d_nonunit_stride [MKLD-13293] source/reproducers/openmp/SimpleCase.CL_INVALID_KERNEL_NAME:SimpleCase.CL_INVALID_KERNEL_NAME_kokkos_test P0 source/reproducers/openmp/class_example:class_example_dyn [CMPLRLLVM-28158] source/reproducers/openmp/enter_data_infinity:enter_data_infinity_static_ath_run [CMPLRLLVM-32204] source/reproducers/openmp/gamess_mini_jit [XDEPS-2759] source/reproducers/openmp/large_transfer_incorect:large_transfer_incorect_large_flag [XDEPS-2757] P0 source/reproducers/openmp/linking_asm [CMPLRLLVM-31189] source/reproducers/openmp/memcopy_multidevice source/reproducers/openmp/mo_gas_concentrations_missing_feature [CMPLRLLVM-29385] source/reproducers/openmp/more_example:more_example_all [CMPLRLLVM-21986] P0 source/reproducers/openmp/one_reduction:one_reduction_complex_ath_run [CMPLRLLVM-22031,XDEPS-3875] P0 source/reproducers/openmp/one_reduction:one_reduction_custom_ath_run [CMPLRLLVM-22031,XDEPS-3875] P0 source/reproducers/openmp/one_reduction:one_reduction_hierarchical_ath_run [CMPLRLLVM-22031,XDEPS-3875] P0 source/reproducers/openmp/slow_reduction source/reproducers/openmp/slow_zeCommandQueueExecuteCommandLists:slow_zeCommandQueueExecuteCommandLists_ath_run [CMPLRLLVM-32940,XDEPS-3448,XDEPS-3829] P0 source/reproducers/openmp/slow_zeCommandQueueExecuteCommandLists:slow_zeCommandQueueExecuteCommandLists_hanging_ath_run [CMPLRLLVM-32940,XDEPS-3448,XDEPS-3829] P0 source/reproducers/openmp/sqrt_simd_issue:sqrt_simd_issue_ath_run [XDEPS-3101] P0 source/reproducers/openmp/static_cl_invalid_kernel_name [CMPLRLLVM-21986] source/reproducers/openmp/target_in_lib [CMPLRLLVM-21986] ## Regressions: source/reproducers/dpcpp/hang_atomic_on_local:hang_atomic_on_local-native_affmask_1_128_ath_run source/reproducers/hybrid/reduction:reduction_sycl [CMPLRLLVM-30136] source/reproducers/ifx/qe_fftxlib_test_crash:qe_fftxlib_test_crash_fftxlib_test [CMPLRLLVM-24637] source/reproducers/mkl/batched_dgemm_hang source/reproducers/openmp/VirtualFunction [CMPLRLLVM-27282] P0 source/reproducers/openmp/check_clang source/reproducers/openmp/global_math:global_math_ath_compile [CMPLRLLVM-31135] source/reproducers/openmp/global_math:global_math_ath_run [CMPLRLLVM-31135] source/reproducers/openmp/l0_follow_the_spec P0 source/reproducers/openmp/map_to_many_sync:map_to_many_sync_with_correct_env [CMPLRLLVM-32920] source/reproducers/openmp/miniqmc-smallrun [CMPLRLLVM-27865] P0 source/reproducers/openmp/static_global_funct:static_global_funct_ath_compile [CMPLRLLVM-28570] source/reproducers/openmp/static_global_funct:static_global_funct_ath_run [CMPLRLLVM-28570] ________________________________ ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, April 20, 2022 9:18 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Upcoming Engineering Compiler and UMD updates At 12pm CDT the oneapi modulefiles will be updated to use the latest version of the User Mode Driver (agama-prerelease-402). These will be made default at 12pm CDT this Thursday (4/21). oneapi/release/2022.01.30.004 oneapi/eng-compiler/2022.01.30.005 Additionally there will be an new engineering compiler which contains bug fixes. Unfortunately this will not be made default due some regressions, which Colleen will send mail about. oneapi/eng-compiler/2022.01.30.006 Modulefiles are active now and defaults will be switched tomorrow. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]>

1 0

New Aurora Compiler+MKL available in JLSE
by Chan-nui, Christopher 21 Apr '22

21 Apr '22

The new modulefiles with the agama-prerelease-402 UMD are no default. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]>

1 0

Upcoming Engineering Compiler and UMD updates
by Chan-nui, Christopher 20 Apr '22

20 Apr '22

At 12pm CDT the oneapi modulefiles will be updated to use the latest version of the User Mode Driver (agama-prerelease-402). These will be made default at 12pm CDT this Thursday (4/21). oneapi/release/2022.01.30.004 oneapi/eng-compiler/2022.01.30.005 Additionally there will be an new engineering compiler which contains bug fixes. Unfortunately this will not be made default due some regressions, which Colleen will send mail about. oneapi/eng-compiler/2022.01.30.006 Modulefiles are active now and defaults will be switched tomorrow. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]>

1 0

Publishing/Presenting NDA work
by Knight, Christopher J. 19 Apr '22

19 Apr '22

Hi all, With the Exascale Computing Project Annual Meeting around the corner, we wanted to send a quick reminder that presentations involving NDA material need to acquire the proper approval. A summary of the process is here on JLSE’s confluence page: https://wiki.jlse.anl.gov/display/inteldga/Publishing+NDA+Work . Some excerpts are below. Please note that any results that make use of the Aurora SDK, even if on the Iris Gen9 nodes, needs to be reviewed before publicly discussed. Please review the process and check if your work involving Intel hardware & software needs to be reviewed. If you are organizing a session at the meeting, please encourage your contributors to review their slides for potential NDA content. Folks should plan on submitting drafts of results they want to present soon considering ECP is asking for slides to be uploaded by April 25th. chris ===== Define what hardware and software is considered to be relevant to this process: * * Should be checked before publishing: * Non-public Intel hardware: DG1 and ATS anywhere and any Intel DevCloud hardware * The Aurora SDK: any form of the Aurora SDK or oneAPI beta SDK software components originating from Intel * Does not need to be cleared: * Results on publicly available Intel hardware (Gen 9, Gen 11) * Results from officially released Intel software or from non-Intel software (Public Intel OpenCL, non-Intel SYCL implementation, …) Categories: Two broad categories of publications from the Intel perspective: 1. Pre-release software (beta software) on public hardware 2. Pre-release hardware * performance results * features that Intel hasn’t announced yet 3. #1 is easier than #2.

1 0

Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler
by Bertoni, Colleen 05 Apr '22

05 Apr '22

Thanks a lot! We'll give this a try, and then submit it to Intel. We'll CC you once we do. Colleen ________________________________ From: Arndt, Daniel <arndtd(a)ornl.gov> Sent: Tuesday, April 5, 2022 5:16 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Bertoni, Colleen <bertoni(a)anl.gov>; Arndt, Daniel <arndtd(a)ornl.gov>; stamoor(a)sandia.gov <stamoor(a)sandia.gov> Subject: RE: Large slowdown with latest oneapi compiler Thanks Colleen! The reproducer can be found in /soft/restricted/CNDA/anl_testing/source/applications/lammps. 1.) Build Kokkos by running `bash build_kokkos.sh` in the kokkos subdirectory (or using an external `Kokkos` in which case the following script needs to be modified accordingly). 2.) Build LAMMPS by running `bash build_sycl.sh` in the lammps_kk_small/src subdirectory 3.) Run the benchmark via `bash run_sycl.sh` in the EXAALT_KPP subdirectory. I tried the newer runtimes 352, 366, and 382 with eng-compiler/latest as well and they all show the slowdown as compared to runtime 316. Running with iprof, it seems that the number of work groups/work items for the kernel stayed the same, though. Best, Daniel Sent from Hiri<https://www.hiri.com/> On 2022-04-05 15:16:56-04:00 Bertoni, Colleen wrote: Thanks a lot for trying! Is the code somewhere that we can reproduce and file a performance bug with Intel? One other thing is that there are several newer runtimes on JLSE as well, so you can try those to see if the performance regression was fixed in a newer version. You can get the newest runtime with the compiler with `module load oneapi/eng-compiler/latest`. (The default version in the module is the newest one which was also stable at the time we did the SDK drop.) Changing the runtime also changes the JIT compiler version which compiles the offload code from SPIRV to assembly, so it could be that the code generation is worse or something like the workgroup size changed. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Arndt, Daniel via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, April 5, 2022 1:35 PM To: stamoor(a)sandia.gov <stamoor(a)sandia.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler I can confirm that the performance issue goes away when using an older runtime, i.e. oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-316 is comparable to oneapi/eng-compiler/2022.01.30.002+intel_compute_runtime/release/agama-prerelease-316 but the performance drops significantly for oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-335 which is the default. Sent from Hiri On 2022-03-30 11:11:36-04:00 Ecp-aurora-sdk-discuss wrote: Thanks for the report! For performance issues normally we want to try to figure out whether the regression is coming from the compiler or the runtime or somewhere else. So you can check a couple of things: --Check if the number of work groups/work items the runtime uses has changed for that kernel. You can do that with `iprof -k ./a.out...` and look at the number of work groups/work items for the kernel. --Try using the current SDK but with the runtime from the last SDK and vice versa (for example, module load oneapi; module switch -f intel_compute_runtime/release/agama-prerelease-316), and maybe we can pinpoint if it's from the compiler or the runtime. Other people on the list might have ideas, but this is where I'd start. If you have time, we can have a call and look at it together. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Moore, Stan via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, March 30, 2022 9:56 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler Hi everyone, I'm seeing a ~6x overall slowdown for the EXAALT KPP benchmark (LAMMPS SNAP) when switching from oneapi/eng-compiler/2022.01.30.002 to oneapi/eng-compiler/2022.01.30.003. This is using the Kokkos SYCL backend, keeping both Kokkos and LAMMPS versions constant. I profiled the kernels and the most expensive kernel (ComputeYi) became ~10x slower due to the compiler update. Most of the other kernels have similar performance between the two compiler versions. I have set SYCL_PROGRAM_COMPILE_OPTIONS="-ze-opt-large-register-file" but it doesn't seem to have any effect. Any ideas? Thank you, Stan

1 0

Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler
by Arndt, Daniel 05 Apr '22

05 Apr '22

Thanks Colleen! The reproducer can be found in /soft/restricted/CNDA/anl_testing/source/applications/lammps. 1.) Build Kokkos by running `bash build_kokkos.sh` in the kokkos subdirectory (or using an external `Kokkos` in which case the following script needs to be modified accordingly). 2.) Build LAMMPS by running `bash build_sycl.sh` in the lammps_kk_small/src subdirectory 3.) Run the benchmark via `bash run_sycl.sh` in the EXAALT_KPP subdirectory. I tried the newer runtimes 352, 366, and 382 with eng-compiler/latest as well and they all show the slowdown as compared to runtime 316. Running with iprof, it seems that the number of work groups/work items for the kernel stayed the same, though. Best, Daniel Sent from Hiri<https://www.hiri.com/> On 2022-04-05 15:16:56-04:00 Bertoni, Colleen wrote: Thanks a lot for trying! Is the code somewhere that we can reproduce and file a performance bug with Intel? One other thing is that there are several newer runtimes on JLSE as well, so you can try those to see if the performance regression was fixed in a newer version. You can get the newest runtime with the compiler with `module load oneapi/eng-compiler/latest`. (The default version in the module is the newest one which was also stable at the time we did the SDK drop.) Changing the runtime also changes the JIT compiler version which compiles the offload code from SPIRV to assembly, so it could be that the code generation is worse or something like the workgroup size changed. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Arndt, Daniel via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, April 5, 2022 1:35 PM To: stamoor(a)sandia.gov <stamoor(a)sandia.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler I can confirm that the performance issue goes away when using an older runtime, i.e. oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-316 is comparable to oneapi/eng-compiler/2022.01.30.002+intel_compute_runtime/release/agama-prerelease-316 but the performance drops significantly for oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-335 which is the default. Sent from Hiri<hxxps://www.hiri.com/> On 2022-03-30 11:11:36-04:00 Ecp-aurora-sdk-discuss wrote: Thanks for the report! For performance issues normally we want to try to figure out whether the regression is coming from the compiler or the runtime or somewhere else. So you can check a couple of things: --Check if the number of work groups/work items the runtime uses has changed for that kernel. You can do that with `iprof -k ./a.out...` and look at the number of work groups/work items for the kernel. --Try using the current SDK but with the runtime from the last SDK and vice versa (for example, module load oneapi; module switch -f intel_compute_runtime/release/agama-prerelease-316), and maybe we can pinpoint if it's from the compiler or the runtime. Other people on the list might have ideas, but this is where I'd start. If you have time, we can have a call and look at it together. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Moore, Stan via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, March 30, 2022 9:56 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler Hi everyone, I'm seeing a ~6x overall slowdown for the EXAALT KPP benchmark (LAMMPS SNAP) when switching from oneapi/eng-compiler/2022.01.30.002 to oneapi/eng-compiler/2022.01.30.003. This is using the Kokkos SYCL backend, keeping both Kokkos and LAMMPS versions constant. I profiled the kernels and the most expensive kernel (ComputeYi) became ~10x slower due to the compiler update. Most of the other kernels have similar performance between the two compiler versions. I have set SYCL_PROGRAM_COMPILE_OPTIONS="-ze-opt-large-register-file" but it doesn't seem to have any effect. Any ideas? Thank you, Stan

1 0

Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler
by Bertoni, Colleen 05 Apr '22

05 Apr '22

Thanks a lot for trying! Is the code somewhere that we can reproduce and file a performance bug with Intel? One other thing is that there are several newer runtimes on JLSE as well, so you can try those to see if the performance regression was fixed in a newer version. You can get the newest runtime with the compiler with `module load oneapi/eng-compiler/latest`. (The default version in the module is the newest one which was also stable at the time we did the SDK drop.) Changing the runtime also changes the JIT compiler version which compiles the offload code from SPIRV to assembly, so it could be that the code generation is worse or something like the workgroup size changed. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Arndt, Daniel via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, April 5, 2022 1:35 PM To: stamoor(a)sandia.gov <stamoor(a)sandia.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler I can confirm that the performance issue goes away when using an older runtime, i.e. oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-316 is comparable to oneapi/eng-compiler/2022.01.30.002+intel_compute_runtime/release/agama-prerelease-316 but the performance drops significantly for oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-335 which is the default. Sent from Hiri<https://www.hiri.com/> On 2022-03-30 11:11:36-04:00 Ecp-aurora-sdk-discuss wrote: Thanks for the report! For performance issues normally we want to try to figure out whether the regression is coming from the compiler or the runtime or somewhere else. So you can check a couple of things: --Check if the number of work groups/work items the runtime uses has changed for that kernel. You can do that with `iprof -k ./a.out...` and look at the number of work groups/work items for the kernel. --Try using the current SDK but with the runtime from the last SDK and vice versa (for example, module load oneapi; module switch -f intel_compute_runtime/release/agama-prerelease-316), and maybe we can pinpoint if it's from the compiler or the runtime. Other people on the list might have ideas, but this is where I'd start. If you have time, we can have a call and look at it together. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Moore, Stan via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, March 30, 2022 9:56 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler Hi everyone, I'm seeing a ~6x overall slowdown for the EXAALT KPP benchmark (LAMMPS SNAP) when switching from oneapi/eng-compiler/2022.01.30.002 to oneapi/eng-compiler/2022.01.30.003. This is using the Kokkos SYCL backend, keeping both Kokkos and LAMMPS versions constant. I profiled the kernels and the most expensive kernel (ComputeYi) became ~10x slower due to the compiler update. Most of the other kernels have similar performance between the two compiler versions. I have set SYCL_PROGRAM_COMPILE_OPTIONS="-ze-opt-large-register-file" but it doesn't seem to have any effect. Any ideas? Thank you, Stan

1 0

Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler
by Arndt, Daniel 05 Apr '22

05 Apr '22

I can confirm that the performance issue goes away when using an older runtime, i.e. oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-316 is comparable to oneapi/eng-compiler/2022.01.30.002+intel_compute_runtime/release/agama-prerelease-316 but the performance drops significantly for oneapi/eng-compiler/2022.01.30.003+intel_compute_runtime/release/agama-prerelease-335 which is the default. Sent from Hiri<https://www.hiri.com/> On 2022-03-30 11:11:36-04:00 Ecp-aurora-sdk-discuss wrote: Thanks for the report! For performance issues normally we want to try to figure out whether the regression is coming from the compiler or the runtime or somewhere else. So you can check a couple of things: --Check if the number of work groups/work items the runtime uses has changed for that kernel. You can do that with `iprof -k ./a.out...` and look at the number of work groups/work items for the kernel. --Try using the current SDK but with the runtime from the last SDK and vice versa (for example, module load oneapi; module switch -f intel_compute_runtime/release/agama-prerelease-316), and maybe we can pinpoint if it's from the compiler or the runtime. Other people on the list might have ideas, but this is where I'd start. If you have time, we can have a call and look at it together. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Moore, Stan via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, March 30, 2022 9:56 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler Hi everyone, I'm seeing a ~6x overall slowdown for the EXAALT KPP benchmark (LAMMPS SNAP) when switching from oneapi/eng-compiler/2022.01.30.002 to oneapi/eng-compiler/2022.01.30.003. This is using the Kokkos SYCL backend, keeping both Kokkos and LAMMPS versions constant. I profiled the kernels and the most expensive kernel (ComputeYi) became ~10x slower due to the compiler update. Most of the other kernels have similar performance between the two compiler versions. I have set SYCL_PROGRAM_COMPILE_OPTIONS="-ze-opt-large-register-file" but it doesn't seem to have any effect. Any ideas? Thank you, Stan

1 0