March 2026 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] MPICH with SYCL on Aurora
by Wozniak, Justin M. 31 Mar '26

31 Mar '26

I should have mentioned- I configured with --with-ze=no to avoid this build-time error: CC src/backend/ze/pup/yaksuri_zei_get_ptr_attr.lo OCLOC (spirv) src/backend/ze/pup/yaksuri_zei_pup_char.cl Could not determine device target: skl. Error: Cannot get HW Info for device skl. Invalid device error, trying to fallback to former ocloc libocloc_legacy1.so Couldn't load former ocloc libocloc_legacy1.so Command was: ocloc compile -file src/backend/ze/pup/yaksuri_zei_pup_char.cl -device skl -spv_only -out_dir src/backend/ze/pup -output_no_suffix -q -options "-I ./src/backend/ze/include -cl-std=CL2.0" make[2]: *** [Makefile:13671: src/backend/ze/pup/yaksuri_zei_pup_char.c] Error 223 make[2]: Leaving directory '/tmp/wozniak/mpich-5.0.0rc3-ze/src/mpi/datatype/typerep/yaksa' Maybe this is the main thing I need to address? Do I just need to force the ocloc device for Aurora? -- Justin M Wozniak ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Tuesday, March 31, 2026 12:02 To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora This is mpich-5.0.0rc3 , I will try that, thanks. -- Justin M Wozniak ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Tuesday, March 31, 2026 11:30 To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G…>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild. Ken From: Wozniak, Justin M. via devel <devel(a)mpich.org> Date: Tuesday, March 31, 2026 at 11:27 AM To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like: Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack: internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed MPIR_Waitall(916)......: MPIDI_IPC_rndv_cb(172).: MPIDI_CMA_copy_data(54): copy_iovs(202).........: process_vm_readv failed (errno 14) Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack: (same) This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time. The app looks like: $ ldd =agent libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000) libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000) libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000) (etc.) libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000) libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000) libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000) libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000) libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000) libc.so.6 => /lib64/libc.so.6 (0x000014670852a000) libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000) (etc.) Thanks -- Justin M Wozniak ________________________________ From: Harms, Kevin <harms(a)alcf.anl.gov> Sent: Monday, March 30, 2026 14:50 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Justin, can you provide the specific error? kevin ________________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, March 30, 2026 2:24 PM To: devel(a)mpich.org Cc: Wozniak, Justin M. Subject: [mpich-devel] MPICH with SYCL on Aurora Hi I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL? Thanks -- Justin M Wozniak

1 0

Re: [mpich-devel] MPICH with SYCL on Aurora
by Wozniak, Justin M. 31 Mar '26

31 Mar '26

This is mpich-5.0.0rc3 , I will try that, thanks. -- Justin M Wozniak ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Tuesday, March 31, 2026 11:30 To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… ). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild. Ken From: Wozniak, Justin M. via devel <devel(a)mpich.org> Date: Tuesday, March 31, 2026 at 11:27 AM To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like: Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack: internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed MPIR_Waitall(916)......: MPIDI_IPC_rndv_cb(172).: MPIDI_CMA_copy_data(54): copy_iovs(202).........: process_vm_readv failed (errno 14) Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack: (same) This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time. The app looks like: $ ldd =agent libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000) libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000) libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000) (etc.) libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000) libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000) libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000) libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000) libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000) libc.so.6 => /lib64/libc.so.6 (0x000014670852a000) libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000) (etc.) Thanks -- Justin M Wozniak ________________________________ From: Harms, Kevin <harms(a)alcf.anl.gov> Sent: Monday, March 30, 2026 14:50 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Justin, can you provide the specific error? kevin ________________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, March 30, 2026 2:24 PM To: devel(a)mpich.org Cc: Wozniak, Justin M. Subject: [mpich-devel] MPICH with SYCL on Aurora Hi I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL? Thanks -- Justin M Wozniak

1 0

Re: [mpich-devel] MPICH with SYCL on Aurora
by Raffenetti, Ken 31 Mar '26

31 Mar '26

Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… ). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild. Ken From: Wozniak, Justin M. via devel <devel(a)mpich.org> Date: Tuesday, March 31, 2026 at 11:27 AM To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like: Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack: internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed MPIR_Waitall(916)......: MPIDI_IPC_rndv_cb(172).: MPIDI_CMA_copy_data(54): copy_iovs(202).........: process_vm_readv failed (errno 14) Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack: (same) This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time. The app looks like: $ ldd =agent libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000) libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000) libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000) (etc.) libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000) libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000) libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000) libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000) libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000) libc.so.6 => /lib64/libc.so.6 (0x000014670852a000) libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000) (etc.) Thanks -- Justin M Wozniak ________________________________ From: Harms, Kevin <harms(a)alcf.anl.gov> Sent: Monday, March 30, 2026 14:50 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Justin, can you provide the specific error? kevin ________________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, March 30, 2026 2:24 PM To: devel(a)mpich.org Cc: Wozniak, Justin M. Subject: [mpich-devel] MPICH with SYCL on Aurora Hi I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL? Thanks -- Justin M Wozniak

1 0

Re: [mpich-devel] MPICH with SYCL on Aurora
by Wozniak, Justin M. 31 Mar '26

31 Mar '26

With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like: Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack: internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed MPIR_Waitall(916)......: MPIDI_IPC_rndv_cb(172).: MPIDI_CMA_copy_data(54): copy_iovs(202).........: process_vm_readv failed (errno 14) Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack: (same) This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time. The app looks like: $ ldd =agent libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000) libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000) libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000) (etc.) libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000) libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000) libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000) libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000) libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000) libc.so.6 => /lib64/libc.so.6 (0x000014670852a000) libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000) (etc.) Thanks -- Justin M Wozniak ________________________________ From: Harms, Kevin <harms(a)alcf.anl.gov> Sent: Monday, March 30, 2026 14:50 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Justin, can you provide the specific error? kevin ________________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, March 30, 2026 2:24 PM To: devel(a)mpich.org Cc: Wozniak, Justin M. Subject: [mpich-devel] MPICH with SYCL on Aurora Hi I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL? Thanks -- Justin M Wozniak

1 0

MPICH 5.0.1rc1 released
by Raffenetti, Ken 31 Mar '26

31 Mar '26

A new release candidate of MPICH, 5.0.1rc1, is now available for download. This release addresses several user-reported issues. For the full set of commits see https://urldefense.us/v3/__https://github.com/pmodels/mpich/compare/v5.0.0_… …v5.0.1rc1/. You can find the release on our downloads page (https://urldefense.us/v3/__https://www.mpich.org/downloads/__;!!G_uCfscf7eW… ) Regards, The MPICH team =============================================================================== Changes in 5.0.1 =============================================================================== # Fix bad cast in release-gather collectives that caused data loss issues on Big-Endian 64b arches (s390x) # Fix issue with canceling MPI_ANY_SOURCE receive requests # Fix configuration issue when C++ compiler does not support complex types # Fix function signature issue in Hydra PBS support # Fix crash in MPI_Allreduce with MPI_LOGICAL type # Fix potential crash in multi-nic libfabric initialization # Fix memory leaks in Level Zero and PMIx support # Fix bug in CMA code when GPU support is enabled # Add large count and other necessary aliases to ROMIO to avoid accidental profiling of internal MPI function usage # Add missing error checks in rndv and colletive composition code # Improve autogen.sh error message when autotools are too old

1 0

Re: [mpich-devel] MPICH with SYCL on Aurora
by Harms, Kevin 30 Mar '26

30 Mar '26

Justin, can you provide the specific error? kevin ________________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, March 30, 2026 2:24 PM To: devel(a)mpich.org Cc: Wozniak, Justin M. Subject: [mpich-devel] MPICH with SYCL on Aurora Hi I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL? Thanks -- Justin M Wozniak

1 0

MPICH with SYCL on Aurora
by Wozniak, Justin M. 30 Mar '26

30 Mar '26

Hi I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL? Thanks -- Justin M Wozniak

1 0

MPICH development call cancelled for the next 2 weeks
by Zhou, Hui 19 Mar '26

19 Mar '26

Dear MPICH family and friends, As I mentioned during today’s call, we are cancelling for the next 2 weeks the MPICH development call due to personal vacations. We’ll resume on 4/9. Enjoy the spring! Thanks, -- Hui Zhou Principal Research Software Engineer Mathematics and Computer Science Argonne National Laboratory TEL: 630-252-3430

1 0

MPICH development call tomorrow
by Zhou, Hui 18 Mar '26

18 Mar '26

Dear MPICH family and friends, The next MPICH development tele-con will be tomorrow (Thursday) at 9am US Central time. agenda * Circulant graph algorithm and a pipelining collective scheduler * Recently merged PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * Recently opened PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * New issues<https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues?q=is*3Ai… > To join the Meeting: https://urldefense.us/v3/__https://teams.microsoft.com/l/meetup-join/19*3am… Thanks, -- Hui Zhou Principal Research Software Engineer Mathematics and Computer Science Argonne National Laboratory TEL: 630-252-3430

1 0

MPICH development call tomorrow
by Zhou, Hui 11 Mar '26

11 Mar '26

Dear MPICH family and friends, The next MPICH development tele-con will be tomorrow (Thursday) at 9am US Central time. agenda * Clean up symbols in ROMIO * Fix ch4 CMP IPC issue * Use template in yaska CUDA kernels * Extend CGA for nonblocking pipelining collectives * Recently merged PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * Recently opened PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * New issues<https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues?q=is*3Ai… > To join the Meeting: https://urldefense.us/v3/__https://teams.microsoft.com/l/meetup-join/19*3am… Thanks, -- Hui Zhou Principal Research Software Engineer Mathematics and Computer Science Argonne National Laboratory TEL: 630-252-3430

1 0