Hi Wyatt--

>     26        /usr/bin/ld: warning: libze_loader.so.1, needed by /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpi.so, not found (try using -rpath or -rpath-link)
>  >> 27    /usr/bin/ld: /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpi.so: undefined reference to `zeMemGetAllocProperties'
> )
> I have the spack oneapi compiler definition set to load the oneapi module, but no others.

You might try adding the needed path to the 'environment' section of the compiler stanza. 

> Outside of spack I get undefined ze symbols from nm on libmpi.so:
> nm  /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpi.so | grep  zeDeviceGet
>                 U zeDeviceGet
>                 U zeDeviceGetCommandQueueGroupProperties
>                 U zeDeviceGetProperties
>                 U zeDeviceGetSubDevices

This is normal, it just means that these symbols are needed and expected to be defined elsewhere, in this case, as part of level zero. 

> These are defined in: /soft/libraries/intel-level-zero/api_+_loader/v1.2.3-Release-2021.06.01/lib64/libze_loader.so.1
> But I'm not sure why the spack build environment would be unable to find that. The directory seems > to be loaded into the LD_LIBRARY_PATH when the oneapi module is loaded.

Yes, this appears to be the case. You might verify that this path is indeed present by examining the build environment (spack build-env) in which case there may be something deeper going on. 

Adding explicitly to your LD_LIBRARY_PATH in the compiler stanza may be your best bet. 

Cheers,

--Frank

Frank Willmore, Ph.D.
ALCF Operations
Argonne National Laboratory
willmore@anl.gov

From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Wyatt Spear via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Sent: Wednesday, July 28, 2021 3:29 PM
To: ecp-aurora-sdk-discuss@jlse.anl.gov <ecp-aurora-sdk-discuss@jlse.anl.gov>
Subject: [Ecp-aurora-sdk-discuss] Undefined ze* symbols using oneapi's mpi with spack
 
Greetings,

I am not sure how to access the archive for this list so please forgive me if this is a known issue (or user error for that matter).

I'm seeing this with the default oneapi module on yarrow/arcticus (oneapi/2021.04.30.004), using spack to build upcxx (and similar errors for a few other packages) for E4S environment testing.

     576    /usr/bin/ld: warning: libze_loader.so.1, needed by /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpicxx.so, not found (try using -rpath or -rpath-link)
  >> 577    /usr/bin/ld: /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpi.so: undefined reference to `zeMemGetAllocProperties'
...numerous other symbols...

(bonus example from hdf5:

     26        /usr/bin/ld: warning: libze_loader.so.1, needed by /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpi.so, not found (try using -rpath or -rpath-link)
  >> 27    /usr/bin/ld: /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpi.so: undefined reference to `zeMemGetAllocProperties'
)

I have the spack oneapi compiler definition set to load the oneapi module, but no others.

Outside of spack I get undefined ze symbols from nm on libmpi.so:
nm  /soft/restricted/CNDA/mpich/drop41.1/mpich-ofi-sockets-icc-default-ats-drop41/lib/libmpi.so | grep  zeDeviceGet
                 U zeDeviceGet
                 U zeDeviceGetCommandQueueGroupProperties
                 U zeDeviceGetProperties
                 U zeDeviceGetSubDevices

But maybe that is expected?

These are defined in: /soft/libraries/intel-level-zero/api_+_loader/v1.2.3-Release-2021.06.01/lib64/libze_loader.so.1
But I'm not sure why the spack build environment would be unable to find that. The directory seems to be loaded into the LD_LIBRARY_PATH when the oneapi module is loaded.

Thanks for any advice.

Wyatt Spear