December 2021 - Ecp-aurora-sdk-discuss

New version of mpich on JLSE
by Chan-nui, Christopher 17 Dec '21

17 Dec '21

Hello, We are pleased to announce that an update to PAPI version 0.5.0 has been deployed on JLSE. For comments and questions please contact Zhang, Peinan peinan.zhang(a)intel.com<mailto:[email protected]>. Release Notes: PAPI package with intel_gpu component supports metrics collection on Intel GPUs. It can collect metrics on multiple GPU devices and multiple tiles on a single GPU devices. How to use it? module use /soft/restricted/CNDA/modulefiles module add oneapi module switch -f intel_compute_runtime intel_compute_runtime/release/agama-prerelease-246 module load papi/intel/0.5.0 What metrics are supported? Intel GPU metrics are platform specific, need to query from GPU in the system. To get all supported metrics, using command papi_native_avail | grep intel_gpu How to specify GPU device, tile and metrics for collection? Intel GPU performance metrics are grouped as MetricGroup for a certain performance focus, so each metric name contains MetricGroupName. For multi-tile GPU device and/or multiple GPU devices system, metrics are collected per tile per device. So device and tile information is part of modifiers to each metric. metric: <MetricGroupName.MetricName>:device=<id>:tile=<id> MetricGroupName: for each collection, all the metrics must belong to same metric group. Otherwise, the collection will error out. device: device id is [0..n-1] for n GPU devices system. If not specified, it is device 0 by default. tile: tile id is [0..m] for m tile GPU. Data collection: To enable metrics collection, set ZET_ENABLE_METRICS=1 PAPI supports two modes to collect Intel GPU metrics. Time based mode by default, the metrics collection is independent of workload execution. Event questy based mode, which is enabled by setting ZET_ENABLE_API_TRACING_EXP=1. metric collection supports workload with implicit scaling. When workload runs without implicit scaling, it runs on tile0 by default. In this case, the PAPI collection will report zero metric values other tiles except tile 0. When workload runs with implicit scaling enabled, it runs on all tiles. In this case, PAPI will report metric valuses on all tiles. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Hart, William E 15 Dec '21

15 Dec '21

The ldd output is at the very bottom of this thread. It looks like the right libsycl.so library. I'll circle back when I've linked with Tim's Kokkos builds. --Bill From: Muralidharan, Servesh <servesh(a)anl.gov> Sent: Wednesday, December 15, 2021 1:41 PM To: Hart, William E <wehart(a)sandia.gov>; ecp-aurora-sdk-discuss(a)jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue One way to confirm if indeed the library are correctly being linked into your python extension, is to find the specific cpython.*.so library generated from it and do a ldd check on it. You should see libsycl.so as a dependency on that list. Yes, Tim's Kokkos builds are the recommended approach to getting a target for Intel GPUs in JLSE. -- Servesh From: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 12:32 PM To: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: RE: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue I'm building my own Kokkos library. In subsequent discussions with Tim Williams, I learned that the system libraries are built with position independent flags. I'll try building with that next. I'm building the extension using SCONS. I'll confirm if '-fsycl' is being used everywhere, but I'm compiling with dpcpp, which I thought added that flag automatically for me. --Bill From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Wednesday, December 15, 2021 1:25 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue So the base env looks fine. Which kokkos library via modulefile are you loading? or are you building one yourself? What does your python extension config looks like? You would need add `extra_link_args` to specify the plugin be built with the link args, especially `-fsycl` there. -- Servesh From: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 12:13 PM To: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: RE: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Servesh: Here's the output. I am using oneapi. --Bill (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> module list Currently Loaded Modulefiles: 1) spack/linux-opensuse_leap15-x86_64(default) <aL> 7) gcc/10.2.0-gcc-10.2.0-yudlyez <aL> 2) hwloc/2.1.0-gcc-10.2.0-wvcmvls <aL> 8) .igc/release/20210921.1(.igc:default) <aL> 3) libnl/3.3.0-gcc-10.2.0-h7m7thf <aL> 9) intel_compute_runtime/release/agama-prerelease-191(default) <aL> 4) rdma-core/27.1-gcc-10.2.0-g6ufs2h <aL> 10) oneapi/eng-compiler/2021.10.30.003(default:default:oneapi) 5) libfabric/1.12.1-gcc-10.2.0-k3r6nnt <aL> 11) cmake/3.20.2-gcc-10.2.0-v26d3st 6) mpi/aurora_mpich/icc-sockets/41.1(aurora_mpich:default:default) <aL> Key: (symbolic-version) <module-tag> <aL>=auto-loaded (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> show_paths LD_LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib CPATH: /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include/level_zero /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include /soft/libraries/khronos/headers/master-2021.09.09/include /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/include /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/include/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpl/linux/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/include PATH: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin/libfabric /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/bin/libfabric /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/cmake-3.20.2-v26d3stmznkvg5wd5otjbacxkbhjtnu4/bin /soft/restricted/CNDA/tools /soft/libraries/khronos/clinfo/master-13ae34-2020.12.14/bin /soft/libraries/intel-level-zero/compute-samples/2020.08.20 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/bin /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/bin /soft/restricted/CNDA/modulefiles/mpi/aurora_mpich/mpich-extended-wrappers/2021.06.08 /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/advisor/2021.4.0-prerelease/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/vtune/2021.9.0_nda_621007/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/condabin /usr/local/bin /usr/bin /bin /usr/lpp/mmfs/bin /home/ac.wehart/.local/bin /home/ac.wehart/bin (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> echo $LD_LIBRARY_PATH /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric:/soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64:/soft/libraries/khronos/loader/master-2021.09.09/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl:/soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib:/soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib: From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Wednesday, December 15, 2021 1:08 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hi Bill, How are you setting up your env? Could you share info on `module list` and if you are loading oneapi, then could you run `show_paths` and attach LD_LIBRARY_PATH output. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Hart, William E via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 11:58 AM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>, Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue FYI, the following worked: LD_PRELOAD=/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 libtbx.python modules/cctbx_project/simtbx/kokkos/tst_kokkos_lib.py But this seems like a total hack. I suspect that there is a Python-specific issue lurking here, where Python isn't configured to load dependent *.so files. --Bill From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> On Behalf Of Hart, William E via Ecp-aurora-sdk-discuss Sent: Wednesday, December 15, 2021 12:44 PM To: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hmmm. I tried this without luck. I'm linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I've tried this without luck so far...) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I'm seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn't seem to be found while resolving symbols. I'm running a *.so file from Python on Arcitcus, and I'm getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I've confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I've confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Muralidharan, Servesh 15 Dec '21

15 Dec '21

One way to confirm if indeed the library are correctly being linked into your python extension, is to find the specific cpython.*.so library generated from it and do a ldd check on it. You should see libsycl.so as a dependency on that list. Yes, Tim’s Kokkos builds are the recommended approach to getting a target for Intel GPUs in JLSE. -- Servesh From: Hart, William E <wehart(a)sandia.gov> Date: Wednesday, December 15, 2021 at 12:32 PM To: Muralidharan, Servesh <servesh(a)anl.gov>, ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: RE: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue I’m building my own Kokkos library. In subsequent discussions with Tim Williams, I learned that the system libraries are built with position independent flags. I’ll try building with that next. I’m building the extension using SCONS. I’ll confirm if ‘-fsycl’ is being used everywhere, but I’m compiling with dpcpp, which I thought added that flag automatically for me. --Bill From: Muralidharan, Servesh <servesh(a)anl.gov> Sent: Wednesday, December 15, 2021 1:25 PM To: Hart, William E <wehart(a)sandia.gov>; ecp-aurora-sdk-discuss(a)jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue So the base env looks fine. Which kokkos library via modulefile are you loading? or are you building one yourself? What does your python extension config looks like? You would need add `extra_link_args` to specify the plugin be built with the link args, especially `-fsycl` there. -- Servesh From: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 12:13 PM To: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: RE: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Servesh: Here’s the output. I am using oneapi. --Bill (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> module list Currently Loaded Modulefiles: 1) spack/linux-opensuse_leap15-x86_64(default) <aL> 7) gcc/10.2.0-gcc-10.2.0-yudlyez <aL> 2) hwloc/2.1.0-gcc-10.2.0-wvcmvls <aL> 8) .igc/release/20210921.1(.igc:default) <aL> 3) libnl/3.3.0-gcc-10.2.0-h7m7thf <aL> 9) intel_compute_runtime/release/agama-prerelease-191(default) <aL> 4) rdma-core/27.1-gcc-10.2.0-g6ufs2h <aL> 10) oneapi/eng-compiler/2021.10.30.003(default:default:oneapi) 5) libfabric/1.12.1-gcc-10.2.0-k3r6nnt <aL> 11) cmake/3.20.2-gcc-10.2.0-v26d3st 6) mpi/aurora_mpich/icc-sockets/41.1(aurora_mpich:default:default) <aL> Key: (symbolic-version) <module-tag> <aL>=auto-loaded (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> show_paths LD_LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib CPATH: /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include/level_zero /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include /soft/libraries/khronos/headers/master-2021.09.09/include /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/include /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/include/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpl/linux/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/include PATH: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin/libfabric /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/bin/libfabric /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/cmake-3.20.2-v26d3stmznkvg5wd5otjbacxkbhjtnu4/bin /soft/restricted/CNDA/tools /soft/libraries/khronos/clinfo/master-13ae34-2020.12.14/bin /soft/libraries/intel-level-zero/compute-samples/2020.08.20 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/bin /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/bin /soft/restricted/CNDA/modulefiles/mpi/aurora_mpich/mpich-extended-wrappers/2021.06.08 /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/advisor/2021.4.0-prerelease/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/vtune/2021.9.0_nda_621007/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/condabin /usr/local/bin /usr/bin /bin /usr/lpp/mmfs/bin /home/ac.wehart/.local/bin /home/ac.wehart/bin (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> echo $LD_LIBRARY_PATH /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric:/soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64:/soft/libraries/khronos/loader/master-2021.09.09/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl:/soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib:/soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib: From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Wednesday, December 15, 2021 1:08 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hi Bill, How are you setting up your env? Could you share info on `module list` and if you are loading oneapi, then could you run `show_paths` and attach LD_LIBRARY_PATH output. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Hart, William E via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 11:58 AM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>, Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue FYI, the following worked: LD_PRELOAD=/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 libtbx.python modules/cctbx_project/simtbx/kokkos/tst_kokkos_lib.py But this seems like a total hack. I suspect that there is a Python-specific issue lurking here, where Python isn’t configured to load dependent *.so files. --Bill From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> On Behalf Of Hart, William E via Ecp-aurora-sdk-discuss Sent: Wednesday, December 15, 2021 12:44 PM To: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hmmm. I tried this without luck. I’m linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I’ve tried this without luck so far…) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I’m seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn’t seem to be found while resolving symbols. I’m running a *.so file from Python on Arcitcus, and I’m getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I’ve confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I’ve confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Hart, William E 15 Dec '21

15 Dec '21

I'm building my own Kokkos library. In subsequent discussions with Tim Williams, I learned that the system libraries are built with position independent flags. I'll try building with that next. I'm building the extension using SCONS. I'll confirm if '-fsycl' is being used everywhere, but I'm compiling with dpcpp, which I thought added that flag automatically for me. --Bill From: Muralidharan, Servesh <servesh(a)anl.gov> Sent: Wednesday, December 15, 2021 1:25 PM To: Hart, William E <wehart(a)sandia.gov>; ecp-aurora-sdk-discuss(a)jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue So the base env looks fine. Which kokkos library via modulefile are you loading? or are you building one yourself? What does your python extension config looks like? You would need add `extra_link_args` to specify the plugin be built with the link args, especially `-fsycl` there. -- Servesh From: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 12:13 PM To: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: RE: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Servesh: Here's the output. I am using oneapi. --Bill (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> module list Currently Loaded Modulefiles: 1) spack/linux-opensuse_leap15-x86_64(default) <aL> 7) gcc/10.2.0-gcc-10.2.0-yudlyez <aL> 2) hwloc/2.1.0-gcc-10.2.0-wvcmvls <aL> 8) .igc/release/20210921.1(.igc:default) <aL> 3) libnl/3.3.0-gcc-10.2.0-h7m7thf <aL> 9) intel_compute_runtime/release/agama-prerelease-191(default) <aL> 4) rdma-core/27.1-gcc-10.2.0-g6ufs2h <aL> 10) oneapi/eng-compiler/2021.10.30.003(default:default:oneapi) 5) libfabric/1.12.1-gcc-10.2.0-k3r6nnt <aL> 11) cmake/3.20.2-gcc-10.2.0-v26d3st 6) mpi/aurora_mpich/icc-sockets/41.1(aurora_mpich:default:default) <aL> Key: (symbolic-version) <module-tag> <aL>=auto-loaded (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> show_paths LD_LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib CPATH: /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include/level_zero /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include /soft/libraries/khronos/headers/master-2021.09.09/include /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/include /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/include/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpl/linux/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/include PATH: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin/libfabric /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/bin/libfabric /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/cmake-3.20.2-v26d3stmznkvg5wd5otjbacxkbhjtnu4/bin /soft/restricted/CNDA/tools /soft/libraries/khronos/clinfo/master-13ae34-2020.12.14/bin /soft/libraries/intel-level-zero/compute-samples/2020.08.20 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/bin /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/bin /soft/restricted/CNDA/modulefiles/mpi/aurora_mpich/mpich-extended-wrappers/2021.06.08 /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/advisor/2021.4.0-prerelease/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/vtune/2021.9.0_nda_621007/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/condabin /usr/local/bin /usr/bin /bin /usr/lpp/mmfs/bin /home/ac.wehart/.local/bin /home/ac.wehart/bin (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$<mailto:ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$> echo $LD_LIBRARY_PATH /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric:/soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64:/soft/libraries/khronos/loader/master-2021.09.09/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl:/soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib:/soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib: From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Wednesday, December 15, 2021 1:08 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hi Bill, How are you setting up your env? Could you share info on `module list` and if you are loading oneapi, then could you run `show_paths` and attach LD_LIBRARY_PATH output. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Hart, William E via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 11:58 AM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>, Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue FYI, the following worked: LD_PRELOAD=/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 libtbx.python modules/cctbx_project/simtbx/kokkos/tst_kokkos_lib.py But this seems like a total hack. I suspect that there is a Python-specific issue lurking here, where Python isn't configured to load dependent *.so files. --Bill From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> On Behalf Of Hart, William E via Ecp-aurora-sdk-discuss Sent: Wednesday, December 15, 2021 12:44 PM To: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hmmm. I tried this without luck. I'm linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I've tried this without luck so far...) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I'm seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn't seem to be found while resolving symbols. I'm running a *.so file from Python on Arcitcus, and I'm getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I've confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I've confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Muralidharan, Servesh 15 Dec '21

15 Dec '21

So the base env looks fine. Which kokkos library via modulefile are you loading? or are you building one yourself? What does your python extension config looks like? You would need add `extra_link_args` to specify the plugin be built with the link args, especially `-fsycl` there. -- Servesh From: Hart, William E <wehart(a)sandia.gov> Date: Wednesday, December 15, 2021 at 12:13 PM To: Muralidharan, Servesh <servesh(a)anl.gov>, ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: RE: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Servesh: Here’s the output. I am using oneapi. --Bill (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$ module list Currently Loaded Modulefiles: 1) spack/linux-opensuse_leap15-x86_64(default) <aL> 7) gcc/10.2.0-gcc-10.2.0-yudlyez <aL> 2) hwloc/2.1.0-gcc-10.2.0-wvcmvls <aL> 8) .igc/release/20210921.1(.igc:default) <aL> 3) libnl/3.3.0-gcc-10.2.0-h7m7thf <aL> 9) intel_compute_runtime/release/agama-prerelease-191(default) <aL> 4) rdma-core/27.1-gcc-10.2.0-g6ufs2h <aL> 10) oneapi/eng-compiler/2021.10.30.003(default:default:oneapi) 5) libfabric/1.12.1-gcc-10.2.0-k3r6nnt <aL> 11) cmake/3.20.2-gcc-10.2.0-v26d3st 6) mpi/aurora_mpich/icc-sockets/41.1(aurora_mpich:default:default) <aL> Key: (symbolic-version) <module-tag> <aL>=auto-loaded (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$ show_paths LD_LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib CPATH: /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include/level_zero /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include /soft/libraries/khronos/headers/master-2021.09.09/include /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/include /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/include/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpl/linux/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/include PATH: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin/libfabric /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/bin/libfabric /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/cmake-3.20.2-v26d3stmznkvg5wd5otjbacxkbhjtnu4/bin /soft/restricted/CNDA/tools /soft/libraries/khronos/clinfo/master-13ae34-2020.12.14/bin /soft/libraries/intel-level-zero/compute-samples/2020.08.20 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/bin /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/bin /soft/restricted/CNDA/modulefiles/mpi/aurora_mpich/mpich-extended-wrappers/2021.06.08 /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/advisor/2021.4.0-prerelease/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/vtune/2021.9.0_nda_621007/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/condabin /usr/local/bin /usr/bin /bin /usr/lpp/mmfs/bin /home/ac.wehart/.local/bin /home/ac.wehart/bin (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$ echo $LD_LIBRARY_PATH /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric:/soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64:/soft/libraries/khronos/loader/master-2021.09.09/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl:/soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib:/soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib: From: Muralidharan, Servesh <servesh(a)anl.gov> Sent: Wednesday, December 15, 2021 1:08 PM To: Hart, William E <wehart(a)sandia.gov>; ecp-aurora-sdk-discuss(a)jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hi Bill, How are you setting up your env? Could you share info on `module list` and if you are loading oneapi, then could you run `show_paths` and attach LD_LIBRARY_PATH output. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Hart, William E via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 11:58 AM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>, Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue FYI, the following worked: LD_PRELOAD=/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 libtbx.python modules/cctbx_project/simtbx/kokkos/tst_kokkos_lib.py But this seems like a total hack. I suspect that there is a Python-specific issue lurking here, where Python isn’t configured to load dependent *.so files. --Bill From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> On Behalf Of Hart, William E via Ecp-aurora-sdk-discuss Sent: Wednesday, December 15, 2021 12:44 PM To: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hmmm. I tried this without luck. I’m linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I’ve tried this without luck so far…) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I’m seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn’t seem to be found while resolving symbols. I’m running a *.so file from Python on Arcitcus, and I’m getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I’ve confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I’ve confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Hart, William E 15 Dec '21

15 Dec '21

Servesh: Here's the output. I am using oneapi. --Bill (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$ module list Currently Loaded Modulefiles: 1) spack/linux-opensuse_leap15-x86_64(default) <aL> 7) gcc/10.2.0-gcc-10.2.0-yudlyez <aL> 2) hwloc/2.1.0-gcc-10.2.0-wvcmvls <aL> 8) .igc/release/20210921.1(.igc:default) <aL> 3) libnl/3.3.0-gcc-10.2.0-h7m7thf <aL> 9) intel_compute_runtime/release/agama-prerelease-191(default) <aL> 4) rdma-core/27.1-gcc-10.2.0-g6ufs2h <aL> 10) oneapi/eng-compiler/2021.10.30.003(default:default:oneapi) 5) libfabric/1.12.1-gcc-10.2.0-k3r6nnt <aL> 11) cmake/3.20.2-gcc-10.2.0-v26d3st 6) mpi/aurora_mpich/icc-sockets/41.1(aurora_mpich:default:default) <aL> Key: (symbolic-version) <module-tag> <aL>=auto-loaded (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$ show_paths LD_LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib LIBRARY_PATH: /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64 /soft/libraries/khronos/loader/master-2021.09.09/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64 /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib CPATH: /soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/include /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include/level_zero /soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/include /soft/libraries/khronos/headers/master-2021.09.09/include /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/include /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/include/cpu_gpu_dpcpp /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpl/linux/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/include /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/include PATH: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin/libfabric /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/bin/libfabric /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/cmake-3.20.2-v26d3stmznkvg5wd5otjbacxkbhjtnu4/bin /soft/restricted/CNDA/tools /soft/libraries/khronos/clinfo/master-13ae34-2020.12.14/bin /soft/libraries/intel-level-zero/compute-samples/2020.08.20 /soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/bin /soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/bin /soft/restricted/CNDA/modulefiles/mpi/aurora_mpich/mpich-extended-wrappers/2021.06.08 /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/bin /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/dpcpp-ct/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/advisor/2021.4.0-prerelease/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/vtune/2021.9.0_nda_621007/bin64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin/intel64 /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/bin /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/condabin /usr/local/bin /usr/bin /bin /usr/lpp/mmfs/bin /home/ac.wehart/.local/bin /home/ac.wehart/bin (2021.4.0-prerelease) ac.wehart@arcticus11:~/exafel/alcc-recipes/cctbx$ echo $LD_LIBRARY_PATH /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/intelpython/latest/envs/2021.4.0-prerelease/lib/libfabric:/soft/restricted/CNDA/emb/perftools/intel-md/20210921.1/lib64:/soft/libraries/khronos/loader/master-2021.09.09/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/api_+_loader/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-gmmlib/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64:/soft/restricted/CNDA/emb/libraries/intel-level-zero/compute-runtime/20210921.1/lib64/intel-opencl:/soft/restricted/CNDA/emb/compilers/intel-igc/20210921.1/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib:/soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libfabric-1.12.1-k3r6nnt6smzxbkpw6rcum2stctobd2hw/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib64:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/rdma-core-27.1-g6ufs2hbzurtcovpifxnuzq6xvdhndm6/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/libnl-3.3.0-h7m7thfzbpu5mmbnant34owd4senj4i2/lib:/soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/hwloc-2.1.0-wvcmvls57izcjculu4ipakltyq24cgyj/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/gdb/intel64/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/debugger/10.2.4/dep/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dal/2021.4.0-prerelease/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/dnnl/2021.4.0-prerelease/cpu_dpcpp_gpu_dpcpp/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/mkl/20211028/lib/intel64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/x64:/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib: From: Muralidharan, Servesh <servesh(a)anl.gov> Sent: Wednesday, December 15, 2021 1:08 PM To: Hart, William E <wehart(a)sandia.gov>; ecp-aurora-sdk-discuss(a)jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hi Bill, How are you setting up your env? Could you share info on `module list` and if you are loading oneapi, then could you run `show_paths` and attach LD_LIBRARY_PATH output. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Hart, William E via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Wednesday, December 15, 2021 at 11:58 AM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>, Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>, ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue FYI, the following worked: LD_PRELOAD=/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 libtbx.python modules/cctbx_project/simtbx/kokkos/tst_kokkos_lib.py But this seems like a total hack. I suspect that there is a Python-specific issue lurking here, where Python isn't configured to load dependent *.so files. --Bill From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> On Behalf Of Hart, William E via Ecp-aurora-sdk-discuss Sent: Wednesday, December 15, 2021 12:44 PM To: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hmmm. I tried this without luck. I'm linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I've tried this without luck so far...) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I'm seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn't seem to be found while resolving symbols. I'm running a *.so file from Python on Arcitcus, and I'm getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I've confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I've confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Muralidharan, Servesh 15 Dec '21

15 Dec '21

Hi Bill, How are you setting up your env? Could you share info on `module list` and if you are loading oneapi, then could you run `show_paths` and attach LD_LIBRARY_PATH output. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Hart, William E via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Wednesday, December 15, 2021 at 11:58 AM To: Hart, William E <wehart(a)sandia.gov>, Zhang, Junchao <jczhang(a)mcs.anl.gov>, ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue FYI, the following worked: LD_PRELOAD=/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 libtbx.python modules/cctbx_project/simtbx/kokkos/tst_kokkos_lib.py But this seems like a total hack. I suspect that there is a Python-specific issue lurking here, where Python isn’t configured to load dependent *.so files. --Bill From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> On Behalf Of Hart, William E via Ecp-aurora-sdk-discuss Sent: Wednesday, December 15, 2021 12:44 PM To: Zhang, Junchao <jczhang(a)mcs.anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hmmm. I tried this without luck. I’m linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I’ve tried this without luck so far…) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I’m seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn’t seem to be found while resolving symbols. I’m running a *.so file from Python on Arcitcus, and I’m getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I’ve confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I’ve confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Hart, William E 15 Dec '21

15 Dec '21

FYI, the following worked: LD_PRELOAD=/soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 libtbx.python modules/cctbx_project/simtbx/kokkos/tst_kokkos_lib.py But this seems like a total hack. I suspect that there is a Python-specific issue lurking here, where Python isn’t configured to load dependent *.so files. --Bill From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> On Behalf Of Hart, William E via Ecp-aurora-sdk-discuss Sent: Wednesday, December 15, 2021 12:44 PM To: Zhang, Junchao <jczhang(a)mcs.anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue Hmmm. I tried this without luck. I’m linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I’ve tried this without luck so far…) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov<mailto:[email protected]>>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I’m seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn’t seem to be found while resolving symbols. I’m running a *.so file from Python on Arcitcus, and I’m getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I’ve confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I’ve confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: A sycl linking issue
by Hart, William E 15 Dec '21

15 Dec '21

Hmmm. I tried this without luck. I’m linking libraries compiled with C++11 and C++17. Do you think that could be a root cause of this? This error occurs when importing a *.so file into python. So maybe I need to load the symbols from the sycl library first before loading that *.so file? (I’ve tried this without luck so far…) --Bill From: Zhang, Junchao <jczhang(a)mcs.anl.gov> Sent: Tuesday, December 14, 2021 10:34 PM To: Hart, William E <wehart(a)sandia.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] A sycl linking issue I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Hart, William E" <wehart(a)sandia.gov<mailto:[email protected]>> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I’m seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn’t seem to be found while resolving symbols. I’m running a *.so file from Python on Arcitcus, and I’m getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I’ve confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I’ve confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0

Re: [Ecp-aurora-sdk-discuss] A sycl linking issue
by Zhang, Junchao 14 Dec '21

14 Dec '21

I met similar problems. It seems RPATH does not work. I had to set LD_LIBRARY_PATH to solve the problem. Maybe you can try that. Thanks -- Junchao Zhang From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of "Hart, William E via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Reply-To: "Hart, William E" <wehart(a)sandia.gov> Date: Tuesday, December 14, 2021 at 12:06 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] A sycl linking issue Hi, all: I’m seeing an unusual issue loading a shared object library on Arcticus, where a Sycl library doesn’t seem to be found while resolving symbols. I’m running a *.so file from Python on Arcitcus, and I’m getting a missing symbol error. Traceback (most recent call last): File "/home/ac.wehart/exafel/alcc-recipes/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext try: mod = __import__(name) ImportError: /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so: undefined symbol: _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ I’ve confirmed that the libsimtbx_kokkos.so file knows about the sycl library: ldd /home/ac.wehart/exafel/alcc-recipes/cctbx/build/lib/libsimtbx_kokkos.so linux-vdso.so.1 (0x00007fff01949000) libomptbx.so => not found libboost_thread.so.1.74.0 => not found libboost_system.so.1.74.0 => not found libboost_python37.so.1.74.0 => not found libimf.so => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x00007fcfd5c4c000) libm.so.6 => /lib64/libm.so.6 (0x00007fcfd590b000) libscitbx_boost_python.so => not found libcctbx.so => not found libsycl.so.5 => /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 (0x00007fcfd5628000) Further, I’ve confirmed that the missing symbol is in this particular libsycl library: nm -o /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5 | grep GetRang /soft/restricted/CNDA/sdk/2021.10.30.001/oneapi/compiler/pseudo-20211026/compiler/linux/lib/libsycl.so.5:00000000002b7270 T _ZN2cl4sycl7handler24GetRangeRoundingSettingsERmS2_S2_ Any clue what might be going on? --Bill -----Original Appointment----- From: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 11:45 AM To: Muralidharan, Servesh; Hart, William E Subject: [EXTERNAL] CCTBX - ExaFEL Kickoff When: Thursday, December 16, 2021 4:00 PM-5:00 PM (UTC-06:00) Central Time (US & Canada). Where: ________________________________________________________________________________ Microsoft Teams meeting Join on your computer or mobile app Click here to join the meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDA0OWIxN2EtOTZkNi00…> Or call in (audio only) +1 630-556-7958,,480495794#<tel:+16305567958,,480495794#> United States, Big Rock Phone Conference ID: 480 495 794# Find a local number<https://dialin.teams.microsoft.com/99d5311c-737a-4ccf-b5fb-198aeb5c306f?id=…> | Reset PIN<https://mysettings.lync.com/pstnconferencing> Learn More<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=9722f6a0-ea98-4777-…> ________________________________________________________________________________

1 0