June 2022 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Re: [Ecp-aurora-sdk-discuss] Issue with sycl::queue::memcpy copying between cpu-malloc and sycl::malloc_shared
by Bertoni, Colleen 24 Jun '22

24 Jun '22

Thanks a lot for the report! The test case in CMPLRLLVM-37963 had a mistake and was fixed, so likely what you're seeing is different from that. Could you send your reproducer? Either attach it here or we can follow up in a separate email chain if you don't want to send it to the whole list. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Asim YarKhan via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Friday, June 24, 2022 6:20 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Issue with sycl::queue::memcpy copying between cpu-malloc and sycl::malloc_shared The ECP SLATE project is having a problem on the JLSE arcticus hardware that may be the same as reported in the "JLSE Aurora SDK Bug, Issue and Request Tracker" as issue number CMPLRLLVM-37963 This is also the same bug reported publicly in the Intel oneAPI Base Toolkit issue tracker at https://community.intel.com/t5/Intel-oneAPI-Base-Toolkit/sycl-queue-memcpy-… The problem is that SLATE uses sycl::queue::memcpy() to move data between users (c-malloc'ed) memory and device (sycl::malloc_shared) memory. For larger data sizes and for sizes that vary (i.e. double precision matrices of 8000,9000) the memcpy() can han The Intel memcpy documentation says that for a sycl::queue::memcpy() the src and dest must both be USM allocations https://intel.github.io/llvm-docs/doxygen/classcl_1_1sycl_1_1queue.html#a6b… However, the SYCL standard states explicitly that both dest and src may be either host or USM pointers. https://www.khronos.org/registry/SYCL/specs/sycl-2020/html/sycl-2020.html#s… I have a reproducer based on the oneAPI-samples/Libraries/oneMKL/matrix_mul_mkl/ sample, but I believe that this problem is just a replica of the "JLSE Aurora SDK Bug, Issue and Request Tracker" issue number CMPLRLLVM-37963, so I have not submitted it separately. Please inform me if you need/prefer an additional reproducer for this sycl::queue::memcpy () behavior. It would be very important for SLATE to be able to move data from user c-allocation to device allocations. Regards, Asim YarKhan Innovative Computing Lab, University of Tennessee

1 0

Issue with sycl::queue::memcpy copying between cpu-malloc and sycl::malloc_shared
by Asim YarKhan 24 Jun '22

24 Jun '22

The ECP SLATE project is having a problem on the JLSE arcticus hardware that may be the same as reported in the "JLSE Aurora SDK Bug, Issue and Request Tracker" as issue number CMPLRLLVM-37963 This is also the same bug reported publicly in the Intel oneAPI Base Toolkit issue tracker at https://community.intel.com/t5/Intel-oneAPI-Base-Toolkit/sycl-queue-memcpy-… The problem is that SLATE uses sycl::queue::memcpy() to move data between users (c-malloc'ed) memory and device (sycl::malloc_shared) memory. For larger data sizes and for sizes that vary (i.e. double precision matrices of 8000,9000) the memcpy() can han The Intel memcpy documentation says that for a sycl::queue::memcpy() the src and dest must both be USM allocations https://intel.github.io/llvm-docs/doxygen/classcl_1_1sycl_1_1queue.html#a6b… However, the SYCL standard states explicitly that both dest and src may be either host or USM pointers. https://www.khronos.org/registry/SYCL/specs/sycl-2020/html/sycl-2020.html#s… I have a reproducer based on the oneAPI-samples/Libraries/oneMKL/matrix_mul_mkl/ sample, but I believe that this problem is just a replica of the "JLSE Aurora SDK Bug, Issue and Request Tracker" issue number CMPLRLLVM-37963, so I have not submitted it separately. Please inform me if you need/prefer an additional reproducer for this sycl::queue::memcpy () behavior. It would be very important for SLATE to be able to move data from user c-allocation to device allocations. Regards, Asim YarKhan Innovative Computing Lab, University of Tennessee

1 0

Re: [Ecp-aurora-sdk-discuss] Issue with sycl::queue::memcpy copying between cpu-malloc and sycl::malloc_shared
by Kumaran, Kalyan 24 Jun '22

24 Jun '22

correcting email address. On Jun 24, 2022, at 6:07 PM, Asim YarKhan <yarkhan(a)icl.utk.edu<mailto:[email protected]>> wrote: The ECP SLATE project is having a problem on the JLSE arcticus hardware that may be the same as reported in the "JLSE Aurora SDK Bug, Issue and Request Tracker" as issue number CMPLRLLVM-37963 This is also the same bug reported publicly in the Intel oneAPI Base Toolkit issue tracker at https://community.intel.com/t5/Intel-oneAPI-Base-Toolkit/sycl-queue-memcpy-… The problem is that SLATE uses sycl::queue::memcpy() to move data between users (c-malloc'ed) memory and device (sycl::malloc_shared) memory. For larger data sizes and for sizes that vary (i.e. double precision matrices of 8000,9000) the memcpy() can han The Intel memcpy documentation says that for a sycl::queue::memcpy() the src and dest must both be USM allocations https://intel.github.io/llvm-docs/doxygen/classcl_1_1sycl_1_1queue.html#a6b… However, the SYCL standard states explicitly that both dest and src may be either host or USM pointers. https://www.khronos.org/registry/SYCL/specs/sycl-2020/html/sycl-2020.html#s… I have a reproducer based on the oneAPI-samples/Libraries/oneMKL/matrix_mul_mkl/ sample, but I believe that this problem is just a replica of the "JLSE Aurora SDK Bug, Issue and Request Tracker" issue number CMPLRLLVM-37963, so I have not submitted it separately. Please inform me if you need/prefer an additional reproducer for this sycl::queue::memcpy () behavior. It would be very important for SLATE to be able to move data from user c-allocation to device allocations. Regards, Asim YarKhan Innovative Computing Lab, University of Tennessee

1 0

New default Kokkos modules and behavior changes
by Williams, Timothy J. 09 Jun '22

09 Jun '22

Dear Aurora testbed Kokkos users, To go along with the just-announced new default oneapi module (oneapi/eng-compiler/2022.01.30.007), we have made some updates to the prebuilt Kokkos installations and associated modules. 1. The default kokkos modules are now those built with the oneapi/eng-compiler/2022.01.30.007 SDK * kokkos/2022.01.30.007/eng-compiler/sycl_intel * kokkos/2022.01.30.007/eng-compiler/sycl_intel_aot * kokkos/2022.01.30.007/eng-compiler/openmptarget_intel * kokkos/2022.01.30.007/eng-compiler/openmptarget_intel_aot 2. All of these were built with the same kokkos sources, a recent git hash from the develop branch. This is a change for the openmptarget builds, which previously were built from a specially modified and quite old set of kokkos sources. 3. There is one known issue with the default SDK and kokkos using OpenMPTarget, which may impact your kokkos application and cause it to crash. Intel is aware of this, and we hope to get the fix in the next compiler release. 4. The absolute default kokkos module—what you get via module load kokkos—was changed from openmptarget_intel to sycl_intel (still a JIT version), to increase the likelihood of getting something usable in the current environment. If you have feedback on whether you would prefer this change to be permanent, please reply back on this thread or directly to me. 5. The kokkos modules, now and going forward, will fail to load if you have loaded an incompatible version of the oneapi module. When that happens, you should see something like this: (base) jlselogin6(~)> module load oneapi/eng-compiler/2022.01.30.002 . . . (base) jlselogin6(~)> module load kokkos/sycl_intel_aot Kokkos version 2022.01.30.007/sycl_intel_aot is not compatible with OneAPI SDK 2022.01.30.002 ---------------------------------------------------------------------------------- Error! No Kokkos module was loaded! ---------------------------------------------------------------------------------- Loading kokkos/sycl_intel_aot ERROR: Module evaluation aborted (base) jlselogin6(~)> Any problems you encounter with this new setup, please report back here on this list, so others are informed, and I’ll try to address them ASAP. Best regards, Tim ------- Timothy J. Williams, Ph.D. --- Argonne National Laboratory --- tjwilliams(a)anl.gov<mailto:[email protected]> -------

1 0

Re: [Ecp-aurora-sdk-discuss] New Engineering Compiler and UMD update
by Chan-nui, Christopher 09 Jun '22

09 Jun '22

The default oneapi modulefile is now oneapi/eng-compiler/2022.01.30.007. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Wednesday, June 8, 2022 at 7:10 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] New Engineering Compiler and UMD update Thursday at 11am CDT we will make oneapi/eng-compiler/2022.01.30.007 the default oneapi environment on JLSE. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]> From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, May 16, 2022 at 9:26 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] New Engineering Compiler and UMD update A new version of the compiler and update UMD (438) are now available (but not default) on the JLSE cluster. oneapi/eng-compiler/2022.01.30.007 This addresses issues found in the 2022.01.30.006 release. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]>

1 0

Re: [Ecp-aurora-sdk-discuss] New Engineering Compiler and UMD update
by Chan-nui, Christopher 08 Jun '22

08 Jun '22

Thursday at 11am CDT we will make oneapi/eng-compiler/2022.01.30.007 the default oneapi environment on JLSE. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]> From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, May 16, 2022 at 9:26 PM To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] New Engineering Compiler and UMD update A new version of the compiler and update UMD (438) are now available (but not default) on the JLSE cluster. oneapi/eng-compiler/2022.01.30.007 This addresses issues found in the 2022.01.30.006 release. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]>

1 0

New DL Frameworks Release
by Ramer, Rebecca 03 Jun '22

03 Jun '22

We would like to announce availability of a new frameworks module for pytorch and tensorflow for use in the JLSE. This release is based on the 2022.2.0 versions of the Deep Learning Frameworks. These versions of the frameworks run on either ATS-P B0 or PVC A0 dgpu, i.e. arcticus or florentia nodes. The new release is available via: frameworks/2022.06.30.001 Information on how to setup and work with this release can be found here for pytorch: https://wiki.jlse.anl.gov/display/inteldga/2022.2.0+-+Pytorch and here for tensorflow: https://wiki.jlse.anl.gov/display/inteldga/2022.2.0+-+Tensorflow Please let us know of any issues encountered. Rebecca Ramer rebecca.ramer(a)intel.com<mailto:[email protected]>

1 0

Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load
by Jiang, Wei 03 Jun '22

03 Jun '22

Collen, your question actually is what we are going to do. Basically due to the current charm++/NAMD build system: NAMD is written in charm++ so the first step is to build charm++, which temporarily doesn't use dpcpp but clang. We are going to add dpcpp into charm++ build system. Wei ________________________________ From: Bertoni, Colleen <bertoni(a)anl.gov> Sent: Friday, June 3, 2022 9:23 AM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Jiang, Wei <wjiang(a)alcf.anl.gov> Subject: Re: clang compiler & LLVM module load Hi Wei, Ye maintains the llvm modules, so he's the best for answering questions about that. But quick question: Is there a reason you want to use standard LLVM (which isn't part of the Aurora SDK) instead of the Aurora SDK ("module load oneapi") for SYCL coding? ("module load oneapi" will put Intel's LLVM which implements DPC++/SYCL in your path, and uses the "dpcpp" wrapper instead of "clang -fsycl ..." to invoke it.) Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Jiang, Wei via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, June 2, 2022 2:06 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Jiang, Wei <wjiang(a)alcf.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load To avoid the runtime conflicts, I just manually added clang13 binary path to PATH without explicitly loading llvm module, and then passed compilation without previous linking error. However, when I ran the code, I received error "No kernel named .....ffiPKfPKN2cl4sycl3vecIfLi4EEESY_iNSV_IfLi3EEESZ_SZ_PT_S11_PdRNSU_5queueEiiENKUlRNSU_7handlerEE_clES16_EUlNSU_7nd_itemILi3EEEE_ was found -46 (CL_INVALID_KERNEL_NAME)Exception caught at file ...". Similar error occurred to my previous sycl -> cuda code migration with codeplay oneapi (uses clang13). The current clang compiler seems involved with cuda rts. ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Jiang, Wei via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, June 2, 2022 1:43 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load I am asking what is the difference between the current and previous installation of clang/llvm on JLSE. Can anybody help answering? ________________________________ From: Luo, Ye <yeluo(a)anl.gov> Sent: Thursday, June 2, 2022 1:28 PM To: Jiang, Wei <wjiang(a)alcf.anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: clang compiler & LLVM module load When I loaded llvm/release-13.0.1, I see "cuda/11.0.2 " is automatically loaded. But when I "module unload cuda/11.0.2", then module llvm/release-13.0.1 is also unloaded automatically. The cuda module is a dependency of the llvm module and unloading cuda unloads llvm The reason why I am concerned this is because, when I recompiled my sycl code, I got clang++ linking error "...ld: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/mkl/20220311/mkl/lib/intel64/libmkl_sycl.so: undefined reference to `clCreateProgramWithIL". This is an error related to runtime system conflicts. For the same code, previously there was no linking error with clang++. The cuda module doesn't add any path at linking. To understand why the link failure, you will need to extract the exact linking command line and debug the issue.The mkl_sycl.so looks for an OpenCL runtime, the "undefined reference" error indicates an incompatible one is being used. You may add -v to clang++ to figure out which one is being picked up and why. This is why I am asking if any big change of clang compiler/llvm module on JLSE. There is no change in llvm modules. All the llvm modules at /soft/modulefiles/llvm loads a CUDA module. If you succeeded in the past, it must be a different clang/llvm installation. Ye ________________________________ From: Jiang, Wei <wjiang(a)alcf.anl.gov> Sent: Thursday, June 2, 2022 12:34 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: clang compiler & LLVM module load When I loaded llvm/release-13.0.1, I see "cuda/11.0.2 " is automatically loaded. But when I "module unload cuda/11.0.2", then module llvm/release-13.0.1 is also unloaded automatically. The reason why I am concerned this is because, when I recompiled my sycl code, I got clang++ linking error "...ld: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/mkl/20220311/mkl/lib/intel64/libmkl_sycl.so: undefined reference to `clCreateProgramWithIL". This is an error related to runtime system conflicts. For the same code, previously there was no linking error with clang++. This is why I am asking if any big change of clang compiler/llvm module on JLSE. ________________________________ From: Luo, Ye <yeluo(a)anl.gov> Sent: Thursday, June 2, 2022 11:00 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Jiang, Wei <wjiang(a)alcf.anl.gov> Subject: Re: clang compiler & LLVM module load Hi Wei, Hi, I notice on JLSE the clang compiler is put into llvm module recently, and loading llvm module requires cuda dependency and therefore essentially linking cudarts.lib. The red part is not true. Users remain responsible to control the linking. Does it mean the clang compiler is set as a compiler "default" for CUDA rts? Clang is compiler. cudarts.libs are just libraries. libraries have no idea about the compiler. So nothing as "default". In short, even if the llvm module loads cuda module, clang doesn't actively engage CUDA toolchain unless being explicitly requested for example when -fopenmp-targets=nvptx64 or compiling cu source code. Ye

1 0

Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load
by Bertoni, Colleen 03 Jun '22

03 Jun '22

Hi Wei, Ye maintains the llvm modules, so he's the best for answering questions about that. But quick question: Is there a reason you want to use standard LLVM (which isn't part of the Aurora SDK) instead of the Aurora SDK ("module load oneapi") for SYCL coding? ("module load oneapi" will put Intel's LLVM which implements DPC++/SYCL in your path, and uses the "dpcpp" wrapper instead of "clang -fsycl ..." to invoke it.) Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Jiang, Wei via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, June 2, 2022 2:06 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Jiang, Wei <wjiang(a)alcf.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load To avoid the runtime conflicts, I just manually added clang13 binary path to PATH without explicitly loading llvm module, and then passed compilation without previous linking error. However, when I ran the code, I received error "No kernel named .....ffiPKfPKN2cl4sycl3vecIfLi4EEESY_iNSV_IfLi3EEESZ_SZ_PT_S11_PdRNSU_5queueEiiENKUlRNSU_7handlerEE_clES16_EUlNSU_7nd_itemILi3EEEE_ was found -46 (CL_INVALID_KERNEL_NAME)Exception caught at file ...". Similar error occurred to my previous sycl -> cuda code migration with codeplay oneapi (uses clang13). The current clang compiler seems involved with cuda rts. ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Jiang, Wei via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, June 2, 2022 1:43 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load I am asking what is the difference between the current and previous installation of clang/llvm on JLSE. Can anybody help answering? ________________________________ From: Luo, Ye <yeluo(a)anl.gov> Sent: Thursday, June 2, 2022 1:28 PM To: Jiang, Wei <wjiang(a)alcf.anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: clang compiler & LLVM module load When I loaded llvm/release-13.0.1, I see "cuda/11.0.2 " is automatically loaded. But when I "module unload cuda/11.0.2", then module llvm/release-13.0.1 is also unloaded automatically. The cuda module is a dependency of the llvm module and unloading cuda unloads llvm The reason why I am concerned this is because, when I recompiled my sycl code, I got clang++ linking error "...ld: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/mkl/20220311/mkl/lib/intel64/libmkl_sycl.so: undefined reference to `clCreateProgramWithIL". This is an error related to runtime system conflicts. For the same code, previously there was no linking error with clang++. The cuda module doesn't add any path at linking. To understand why the link failure, you will need to extract the exact linking command line and debug the issue.The mkl_sycl.so looks for an OpenCL runtime, the "undefined reference" error indicates an incompatible one is being used. You may add -v to clang++ to figure out which one is being picked up and why. This is why I am asking if any big change of clang compiler/llvm module on JLSE. There is no change in llvm modules. All the llvm modules at /soft/modulefiles/llvm loads a CUDA module. If you succeeded in the past, it must be a different clang/llvm installation. Ye ________________________________ From: Jiang, Wei <wjiang(a)alcf.anl.gov> Sent: Thursday, June 2, 2022 12:34 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: clang compiler & LLVM module load When I loaded llvm/release-13.0.1, I see "cuda/11.0.2 " is automatically loaded. But when I "module unload cuda/11.0.2", then module llvm/release-13.0.1 is also unloaded automatically. The reason why I am concerned this is because, when I recompiled my sycl code, I got clang++ linking error "...ld: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/mkl/20220311/mkl/lib/intel64/libmkl_sycl.so: undefined reference to `clCreateProgramWithIL". This is an error related to runtime system conflicts. For the same code, previously there was no linking error with clang++. This is why I am asking if any big change of clang compiler/llvm module on JLSE. ________________________________ From: Luo, Ye <yeluo(a)anl.gov> Sent: Thursday, June 2, 2022 11:00 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Jiang, Wei <wjiang(a)alcf.anl.gov> Subject: Re: clang compiler & LLVM module load Hi Wei, Hi, I notice on JLSE the clang compiler is put into llvm module recently, and loading llvm module requires cuda dependency and therefore essentially linking cudarts.lib. The red part is not true. Users remain responsible to control the linking. Does it mean the clang compiler is set as a compiler "default" for CUDA rts? Clang is compiler. cudarts.libs are just libraries. libraries have no idea about the compiler. So nothing as "default". In short, even if the llvm module loads cuda module, clang doesn't actively engage CUDA toolchain unless being explicitly requested for example when -fopenmp-targets=nvptx64 or compiling cu source code. Ye

1 0

Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load
by Jiang, Wei 02 Jun '22

02 Jun '22

To avoid the runtime conflicts, I just manually added clang13 binary path to PATH without explicitly loading llvm module, and then passed compilation without previous linking error. However, when I ran the code, I received error "No kernel named .....ffiPKfPKN2cl4sycl3vecIfLi4EEESY_iNSV_IfLi3EEESZ_SZ_PT_S11_PdRNSU_5queueEiiENKUlRNSU_7handlerEE_clES16_EUlNSU_7nd_itemILi3EEEE_ was found -46 (CL_INVALID_KERNEL_NAME)Exception caught at file ...". Similar error occurred to my previous sycl -> cuda code migration with codeplay oneapi (uses clang13). The current clang compiler seems involved with cuda rts. ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Jiang, Wei via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, June 2, 2022 1:43 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] clang compiler & LLVM module load I am asking what is the difference between the current and previous installation of clang/llvm on JLSE. Can anybody help answering? ________________________________ From: Luo, Ye <yeluo(a)anl.gov> Sent: Thursday, June 2, 2022 1:28 PM To: Jiang, Wei <wjiang(a)alcf.anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: clang compiler & LLVM module load When I loaded llvm/release-13.0.1, I see "cuda/11.0.2 " is automatically loaded. But when I "module unload cuda/11.0.2", then module llvm/release-13.0.1 is also unloaded automatically. The cuda module is a dependency of the llvm module and unloading cuda unloads llvm The reason why I am concerned this is because, when I recompiled my sycl code, I got clang++ linking error "...ld: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/mkl/20220311/mkl/lib/intel64/libmkl_sycl.so: undefined reference to `clCreateProgramWithIL". This is an error related to runtime system conflicts. For the same code, previously there was no linking error with clang++. The cuda module doesn't add any path at linking. To understand why the link failure, you will need to extract the exact linking command line and debug the issue.The mkl_sycl.so looks for an OpenCL runtime, the "undefined reference" error indicates an incompatible one is being used. You may add -v to clang++ to figure out which one is being picked up and why. This is why I am asking if any big change of clang compiler/llvm module on JLSE. There is no change in llvm modules. All the llvm modules at /soft/modulefiles/llvm loads a CUDA module. If you succeeded in the past, it must be a different clang/llvm installation. Ye ________________________________ From: Jiang, Wei <wjiang(a)alcf.anl.gov> Sent: Thursday, June 2, 2022 12:34 PM To: Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: clang compiler & LLVM module load When I loaded llvm/release-13.0.1, I see "cuda/11.0.2 " is automatically loaded. But when I "module unload cuda/11.0.2", then module llvm/release-13.0.1 is also unloaded automatically. The reason why I am concerned this is because, when I recompiled my sycl code, I got clang++ linking error "...ld: /soft/restricted/CNDA/sdk/2022.01.30.001/oneapi/mkl/20220311/mkl/lib/intel64/libmkl_sycl.so: undefined reference to `clCreateProgramWithIL". This is an error related to runtime system conflicts. For the same code, previously there was no linking error with clang++. This is why I am asking if any big change of clang compiler/llvm module on JLSE. ________________________________ From: Luo, Ye <yeluo(a)anl.gov> Sent: Thursday, June 2, 2022 11:00 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Jiang, Wei <wjiang(a)alcf.anl.gov> Subject: Re: clang compiler & LLVM module load Hi Wei, Hi, I notice on JLSE the clang compiler is put into llvm module recently, and loading llvm module requires cuda dependency and therefore essentially linking cudarts.lib. The red part is not true. Users remain responsible to control the linking. Does it mean the clang compiler is set as a compiler "default" for CUDA rts? Clang is compiler. cudarts.libs are just libraries. libraries have no idea about the compiler. So nothing as "default". In short, even if the llvm module loads cuda module, clang doesn't actively engage CUDA toolchain unless being explicitly requested for example when -fopenmp-targets=nvptx64 or compiling cu source code. Ye

1 0