A new release candidate of MPICH, 5.0.1rc3, is now available for download. This release addresses a number of user-reported issues, including a potential corruption issue with multiple singleton inits on the same node. For the full set of commits see https://urldefense.us/v3/__https://github.com/pmodels/mpich/compare/v5.0.0_… …v5.0.1rc3/.
You can find the release on our downloads page (https://urldefense.us/v3/__https://www.mpich.org/downloads/__;!!G_uCfscf7eW… )
Regards,
The MPICH team
===============================================================================
Changes in 5.0.1
===============================================================================
# Fix bad cast in release-gather collectives that caused data loss
issues on Big-Endian 64b arches (s390x)
# Fix issue with canceling MPI_ANY_SOURCE receive requests
# Fix configuration issue when C++ compiler does not support complex types
# Fix function signature issue in Hydra PBS support
# Fix crash in MPI_Allreduce with MPI_LOGICAL type
# Fix potential crash in multi-nic libfabric initialization
# Fix memory leaks in Level Zero and PMIx support
# Fix bug in CMA code when GPU support is enabled
# Fix potential shared memory collision between multiple singleton
processes on the same node
# Add large count and other necessary aliases to ROMIO to avoid
accidental profiling of internal MPI function usage
# Add missing error checks in rndv and colletive composition code
# Improve autogen.sh error message when autotools are too old
I see. First, try set MPIR_CVAR_NO_LOCAL=1 and FI_PROVIDER=cxi. This forces the network path. If it doesn’t show the same slowdown, it confirms the issue is in the shared memory communication.
Next try MPIR_CVAR_CH4_XPMEM_ENABLE=0 and MPIR_CVAR_CH4_CMA_ENABLE=0 to ensure that the slow down is not due to these two IPC modules.
Lastly, try set MPIR_CVAR_CH4_IPC_GPU_P2P_THRESHOLD to some big value, e.g. 10000000000, to check whether the slowdown is due to the GPU IPC path not being selected.
--
Hui Zhou
From: Wozniak, Justin M. <woz(a)anl.gov>
Date: Monday, April 6, 2026 at 10:56 AM
To: Zhou, Hui <zhouh(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org>, Raffenetti, Ken <raffenet(a)anl.gov>, Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Compared to the single process run. This is not the case when built with the system MPI.
--
Justin M Wozniak
________________________________
From: Zhou, Hui <zhouh(a)anl.gov>
Sent: Sunday, April 5, 2026 20:27
To: devel(a)mpich.org <devel(a)mpich.org>; Raffenetti, Ken <raffenet(a)anl.gov>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Hi Justin. 6x slower compared to which baseline?
Hui
Get Outlook for Android<https://urldefense.us/v3/__https://aka.ms/AAb9ysg__;!!G_uCfscf7eWS!d783bqeX… >
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Thursday, April 2, 2026 8:32:59 AM
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
That makes it run but is about 6x slower when running with 2 processes. Is there something else I can check? I get:
# --with-device=ch4:ofi
# *** Device Configuration
# *** --------------------
# *** device : ch4:ofi
# *** ipc feature : xpmem gpudirect
# *** gpu support : ZE
Note that I do not need any network support, as I am using this MPICH for node-local communication only. So I also do not need integrated process launch features.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Wednesday, April 1, 2026 9:10
To: Wozniak, Justin M. <woz(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
If building with Intel GPU support on Aurora, you should have --with-ze and --enable-ze-native=pvc in the configuration line. See https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC…> for more details.
Ken
From: Wozniak, Justin M. <woz(a)anl.gov>
Date: Tuesday, March 31, 2026 at 4:28 PM
To: Raffenetti, Ken <raffenet(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org>, Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
I should have mentioned- I configured with --with-ze=no to avoid this build-time error:
CC src/backend/ze/pup/yaksuri_zei_get_ptr_attr.lo
OCLOC (spirv) src/backend/ze/pup/yaksuri_zei_pup_char.cl
Could not determine device target: skl.
Error: Cannot get HW Info for device skl.
Invalid device error, trying to fallback to former ocloc libocloc_legacy1.so
Couldn't load former ocloc libocloc_legacy1.so
Command was: ocloc compile -file src/backend/ze/pup/yaksuri_zei_pup_char.cl -device skl -spv_only -out_dir src/backend/ze/pup -output_no_suffix -q -options "-I ./src/backend/ze/include -cl-std=CL2.0"
make[2]: *** [Makefile:13671: src/backend/ze/pup/yaksuri_zei_pup_char.c] Error 223
make[2]: Leaving directory '/tmp/wozniak/mpich-5.0.0rc3-ze/src/mpi/datatype/typerep/yaksa'
Maybe this is the main thing I need to address? Do I just need to force the ocloc device for Aurora?
--
Justin M Wozniak
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Tuesday, March 31, 2026 12:02
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
This is mpich-5.0.0rc3 , I will try that, thanks.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Tuesday, March 31, 2026 11:30
To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G…>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild.
Ken
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Date: Tuesday, March 31, 2026 at 11:27 AM
To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like:
Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack:
internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed
MPIR_Waitall(916)......:
MPIDI_IPC_rndv_cb(172).:
MPIDI_CMA_copy_data(54):
copy_iovs(202).........: process_vm_readv failed (errno 14)
Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack:
(same)
This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time.
The app looks like:
$ ldd =agent
libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000)
libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000)
libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000)
(etc.)
libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000)
libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000)
libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000)
libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000)
libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000)
libc.so.6 => /lib64/libc.so.6 (0x000014670852a000)
libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000)
(etc.)
Thanks
--
Justin M Wozniak
________________________________
From: Harms, Kevin <harms(a)alcf.anl.gov>
Sent: Monday, March 30, 2026 14:50
To: devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Justin,
can you provide the specific error?
kevin
________________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Monday, March 30, 2026 2:24 PM
To: devel(a)mpich.org
Cc: Wozniak, Justin M.
Subject: [mpich-devel] MPICH with SYCL on Aurora
Hi
I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL?
Thanks
--
Justin M Wozniak
Compared to the single process run. This is not the case when built with the system MPI.
--
Justin M Wozniak
________________________________
From: Zhou, Hui <zhouh(a)anl.gov>
Sent: Sunday, April 5, 2026 20:27
To: devel(a)mpich.org <devel(a)mpich.org>; Raffenetti, Ken <raffenet(a)anl.gov>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Hi Justin. 6x slower compared to which baseline?
Hui
Get Outlook for Android<https://urldefense.us/v3/__https://aka.ms/AAb9ysg__;!!G_uCfscf7eWS!dWLTYWC0… >
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Thursday, April 2, 2026 8:32:59 AM
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
That makes it run but is about 6x slower when running with 2 processes. Is there something else I can check? I get:
# --with-device=ch4:ofi
# *** Device Configuration
# *** --------------------
# *** device : ch4:ofi
# *** ipc feature : xpmem gpudirect
# *** gpu support : ZE
Note that I do not need any network support, as I am using this MPICH for node-local communication only. So I also do not need integrated process launch features.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Wednesday, April 1, 2026 9:10
To: Wozniak, Justin M. <woz(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
If building with Intel GPU support on Aurora, you should have --with-ze and --enable-ze-native=pvc in the configuration line. See https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC…> for more details.
Ken
From: Wozniak, Justin M. <woz(a)anl.gov>
Date: Tuesday, March 31, 2026 at 4:28 PM
To: Raffenetti, Ken <raffenet(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org>, Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
I should have mentioned- I configured with --with-ze=no to avoid this build-time error:
CC src/backend/ze/pup/yaksuri_zei_get_ptr_attr.lo
OCLOC (spirv) src/backend/ze/pup/yaksuri_zei_pup_char.cl
Could not determine device target: skl.
Error: Cannot get HW Info for device skl.
Invalid device error, trying to fallback to former ocloc libocloc_legacy1.so
Couldn't load former ocloc libocloc_legacy1.so
Command was: ocloc compile -file src/backend/ze/pup/yaksuri_zei_pup_char.cl -device skl -spv_only -out_dir src/backend/ze/pup -output_no_suffix -q -options "-I ./src/backend/ze/include -cl-std=CL2.0"
make[2]: *** [Makefile:13671: src/backend/ze/pup/yaksuri_zei_pup_char.c] Error 223
make[2]: Leaving directory '/tmp/wozniak/mpich-5.0.0rc3-ze/src/mpi/datatype/typerep/yaksa'
Maybe this is the main thing I need to address? Do I just need to force the ocloc device for Aurora?
--
Justin M Wozniak
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Tuesday, March 31, 2026 12:02
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
This is mpich-5.0.0rc3 , I will try that, thanks.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Tuesday, March 31, 2026 11:30
To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G…>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild.
Ken
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Date: Tuesday, March 31, 2026 at 11:27 AM
To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like:
Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack:
internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed
MPIR_Waitall(916)......:
MPIDI_IPC_rndv_cb(172).:
MPIDI_CMA_copy_data(54):
copy_iovs(202).........: process_vm_readv failed (errno 14)
Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack:
(same)
This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time.
The app looks like:
$ ldd =agent
libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000)
libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000)
libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000)
(etc.)
libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000)
libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000)
libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000)
libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000)
libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000)
libc.so.6 => /lib64/libc.so.6 (0x000014670852a000)
libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000)
(etc.)
Thanks
--
Justin M Wozniak
________________________________
From: Harms, Kevin <harms(a)alcf.anl.gov>
Sent: Monday, March 30, 2026 14:50
To: devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Justin,
can you provide the specific error?
kevin
________________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Monday, March 30, 2026 2:24 PM
To: devel(a)mpich.org
Cc: Wozniak, Justin M.
Subject: [mpich-devel] MPICH with SYCL on Aurora
Hi
I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL?
Thanks
--
Justin M Wozniak
Hi Justin. 6x slower compared to which baseline?
Hui
Get Outlook for Android<https://urldefense.us/v3/__https://aka.ms/AAb9ysg__;!!G_uCfscf7eWS!eylPXKJS… >
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Thursday, April 2, 2026 8:32:59 AM
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
That makes it run but is about 6x slower when running with 2 processes. Is there something else I can check? I get:
# --with-device=ch4:ofi
# *** Device Configuration
# *** --------------------
# *** device : ch4:ofi
# *** ipc feature : xpmem gpudirect
# *** gpu support : ZE
Note that I do not need any network support, as I am using this MPICH for node-local communication only. So I also do not need integrated process launch features.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Wednesday, April 1, 2026 9:10
To: Wozniak, Justin M. <woz(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
If building with Intel GPU support on Aurora, you should have --with-ze and --enable-ze-native=pvc in the configuration line. See https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC…> for more details.
Ken
From: Wozniak, Justin M. <woz(a)anl.gov>
Date: Tuesday, March 31, 2026 at 4:28 PM
To: Raffenetti, Ken <raffenet(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org>, Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
I should have mentioned- I configured with --with-ze=no to avoid this build-time error:
CC src/backend/ze/pup/yaksuri_zei_get_ptr_attr.lo
OCLOC (spirv) src/backend/ze/pup/yaksuri_zei_pup_char.cl
Could not determine device target: skl.
Error: Cannot get HW Info for device skl.
Invalid device error, trying to fallback to former ocloc libocloc_legacy1.so
Couldn't load former ocloc libocloc_legacy1.so
Command was: ocloc compile -file src/backend/ze/pup/yaksuri_zei_pup_char.cl -device skl -spv_only -out_dir src/backend/ze/pup -output_no_suffix -q -options "-I ./src/backend/ze/include -cl-std=CL2.0"
make[2]: *** [Makefile:13671: src/backend/ze/pup/yaksuri_zei_pup_char.c] Error 223
make[2]: Leaving directory '/tmp/wozniak/mpich-5.0.0rc3-ze/src/mpi/datatype/typerep/yaksa'
Maybe this is the main thing I need to address? Do I just need to force the ocloc device for Aurora?
--
Justin M Wozniak
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Tuesday, March 31, 2026 12:02
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
This is mpich-5.0.0rc3 , I will try that, thanks.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Tuesday, March 31, 2026 11:30
To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G…>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild.
Ken
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Date: Tuesday, March 31, 2026 at 11:27 AM
To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like:
Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack:
internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed
MPIR_Waitall(916)......:
MPIDI_IPC_rndv_cb(172).:
MPIDI_CMA_copy_data(54):
copy_iovs(202).........: process_vm_readv failed (errno 14)
Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack:
(same)
This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time.
The app looks like:
$ ldd =agent
libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000)
libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000)
libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000)
(etc.)
libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000)
libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000)
libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000)
libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000)
libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000)
libc.so.6 => /lib64/libc.so.6 (0x000014670852a000)
libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000)
(etc.)
Thanks
--
Justin M Wozniak
________________________________
From: Harms, Kevin <harms(a)alcf.anl.gov>
Sent: Monday, March 30, 2026 14:50
To: devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Justin,
can you provide the specific error?
kevin
________________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Monday, March 30, 2026 2:24 PM
To: devel(a)mpich.org
Cc: Wozniak, Justin M.
Subject: [mpich-devel] MPICH with SYCL on Aurora
Hi
I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL?
Thanks
--
Justin M Wozniak
That makes it run but is about 6x slower when running with 2 processes. Is there something else I can check? I get:
# --with-device=ch4:ofi
# *** Device Configuration
# *** --------------------
# *** device : ch4:ofi
# *** ipc feature : xpmem gpudirect
# *** gpu support : ZE
Note that I do not need any network support, as I am using this MPICH for node-local communication only. So I also do not need integrated process launch features.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Wednesday, April 1, 2026 9:10
To: Wozniak, Justin M. <woz(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
If building with Intel GPU support on Aurora, you should have --with-ze and --enable-ze-native=pvc in the configuration line. See https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC… for more details.
Ken
From: Wozniak, Justin M. <woz(a)anl.gov>
Date: Tuesday, March 31, 2026 at 4:28 PM
To: Raffenetti, Ken <raffenet(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org>, Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
I should have mentioned- I configured with --with-ze=no to avoid this build-time error:
CC src/backend/ze/pup/yaksuri_zei_get_ptr_attr.lo
OCLOC (spirv) src/backend/ze/pup/yaksuri_zei_pup_char.cl
Could not determine device target: skl.
Error: Cannot get HW Info for device skl.
Invalid device error, trying to fallback to former ocloc libocloc_legacy1.so
Couldn't load former ocloc libocloc_legacy1.so
Command was: ocloc compile -file src/backend/ze/pup/yaksuri_zei_pup_char.cl -device skl -spv_only -out_dir src/backend/ze/pup -output_no_suffix -q -options "-I ./src/backend/ze/include -cl-std=CL2.0"
make[2]: *** [Makefile:13671: src/backend/ze/pup/yaksuri_zei_pup_char.c] Error 223
make[2]: Leaving directory '/tmp/wozniak/mpich-5.0.0rc3-ze/src/mpi/datatype/typerep/yaksa'
Maybe this is the main thing I need to address? Do I just need to force the ocloc device for Aurora?
--
Justin M Wozniak
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Tuesday, March 31, 2026 12:02
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
This is mpich-5.0.0rc3 , I will try that, thanks.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Tuesday, March 31, 2026 11:30
To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G…>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild.
Ken
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Date: Tuesday, March 31, 2026 at 11:27 AM
To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like:
Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack:
internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed
MPIR_Waitall(916)......:
MPIDI_IPC_rndv_cb(172).:
MPIDI_CMA_copy_data(54):
copy_iovs(202).........: process_vm_readv failed (errno 14)
Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack:
(same)
This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time.
The app looks like:
$ ldd =agent
libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000)
libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000)
libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000)
(etc.)
libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000)
libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000)
libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000)
libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000)
libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000)
libc.so.6 => /lib64/libc.so.6 (0x000014670852a000)
libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000)
(etc.)
Thanks
--
Justin M Wozniak
________________________________
From: Harms, Kevin <harms(a)alcf.anl.gov>
Sent: Monday, March 30, 2026 14:50
To: devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Justin,
can you provide the specific error?
kevin
________________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Monday, March 30, 2026 2:24 PM
To: devel(a)mpich.org
Cc: Wozniak, Justin M.
Subject: [mpich-devel] MPICH with SYCL on Aurora
Hi
I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL?
Thanks
--
Justin M Wozniak
A new release candidate of MPICH, 5.0.1rc2, is now available for download. This release addresses several user-reported issues, including an accidental breakage of the standard ABI build (–enable-mpi-abi) present in 5.0.1rc1. For the full set of commits see https://urldefense.us/v3/__https://github.com/pmodels/mpich/compare/v5.0.0_… …v5.0.1rc2/.
You can find the release on our downloads page (https://urldefense.us/v3/__https://www.mpich.org/downloads/__;!!G_uCfscf7eW… )
Regards,
The MPICH team
===============================================================================
Changes in 5.0.1
===============================================================================
# Fix bad cast in release-gather collectives that caused data loss
issues on Big-Endian 64b arches (s390x)
# Fix issue with canceling MPI_ANY_SOURCE receive requests
# Fix configuration issue when C++ compiler does not support complex types
# Fix function signature issue in Hydra PBS support
# Fix crash in MPI_Allreduce with MPI_LOGICAL type
# Fix potential crash in multi-nic libfabric initialization
# Fix memory leaks in Level Zero and PMIx support
# Fix bug in CMA code when GPU support is enabled
# Add large count and other necessary aliases to ROMIO to avoid
accidental profiling of internal MPI function usage
# Add missing error checks in rndv and colletive composition code
# Improve autogen.sh error message when autotools are too old
If building with Intel GPU support on Aurora, you should have --with-ze and --enable-ze-native=pvc in the configuration line. See https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC… for more details.
Ken
From: Wozniak, Justin M. <woz(a)anl.gov>
Date: Tuesday, March 31, 2026 at 4:28 PM
To: Raffenetti, Ken <raffenet(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org>, Harms, Kevin <harms(a)alcf.anl.gov>
Subject: Re: MPICH with SYCL on Aurora
I should have mentioned- I configured with --with-ze=no to avoid this build-time error:
CC src/backend/ze/pup/yaksuri_zei_get_ptr_attr.lo
OCLOC (spirv) src/backend/ze/pup/yaksuri_zei_pup_char.cl
Could not determine device target: skl.
Error: Cannot get HW Info for device skl.
Invalid device error, trying to fallback to former ocloc libocloc_legacy1.so
Couldn't load former ocloc libocloc_legacy1.so
Command was: ocloc compile -file src/backend/ze/pup/yaksuri_zei_pup_char.cl -device skl -spv_only -out_dir src/backend/ze/pup -output_no_suffix -q -options "-I ./src/backend/ze/include -cl-std=CL2.0"
make[2]: *** [Makefile:13671: src/backend/ze/pup/yaksuri_zei_pup_char.c] Error 223
make[2]: Leaving directory '/tmp/wozniak/mpich-5.0.0rc3-ze/src/mpi/datatype/typerep/yaksa'
Maybe this is the main thing I need to address? Do I just need to force the ocloc device for Aurora?
--
Justin M Wozniak
________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Tuesday, March 31, 2026 12:02
To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
This is mpich-5.0.0rc3 , I will try that, thanks.
--
Justin M Wozniak
________________________________
From: Raffenetti, Ken <raffenet(a)anl.gov>
Sent: Tuesday, March 31, 2026 11:30
To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G…>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild.
Ken
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Date: Tuesday, March 31, 2026 at 11:27 AM
To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: [mpich-devel] MPICH with SYCL on Aurora
With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like:
Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack:
internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed
MPIR_Waitall(916)......:
MPIDI_IPC_rndv_cb(172).:
MPIDI_CMA_copy_data(54):
copy_iovs(202).........: process_vm_readv failed (errno 14)
Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack:
(same)
This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time.
The app looks like:
$ ldd =agent
libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000)
libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000)
libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000)
(etc.)
libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000)
libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000)
libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000)
libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000)
libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000)
libc.so.6 => /lib64/libc.so.6 (0x000014670852a000)
libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000)
(etc.)
Thanks
--
Justin M Wozniak
________________________________
From: Harms, Kevin <harms(a)alcf.anl.gov>
Sent: Monday, March 30, 2026 14:50
To: devel(a)mpich.org <devel(a)mpich.org>
Cc: Wozniak, Justin M. <woz(a)anl.gov>
Subject: Re: MPICH with SYCL on Aurora
Justin,
can you provide the specific error?
kevin
________________________________________
From: Wozniak, Justin M. via devel <devel(a)mpich.org>
Sent: Monday, March 30, 2026 2:24 PM
To: devel(a)mpich.org
Cc: Wozniak, Justin M.
Subject: [mpich-devel] MPICH with SYCL on Aurora
Hi
I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL?
Thanks
--
Justin M Wozniak
