April 2026 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job
by Zhou, Hui 27 Apr '26

27 Apr '26

[like] Zhou, Hui reacted to your message: ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, 27 April 2026 17:15:51 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job That worked, thanks! I just had to unset PMIX_NAMESPACE at runtime. -- Justin M Wozniak ________________________________ From: Zhou, Hui via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 13:44 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job I see the issue! The application main.x is linked to a version of MPICH build with PMIx and hydra only uses PMI1 or PMI2. You need rebuild MPICH without PMIx and relink the app with the custom built MPICH. There is a way to build MPICH with both PMIx and PMI1/2, but it is not well tested. Try specify both the configure option --with-pmix=... and --with-pmilib=mpich in building MPICH, and let me know if it works. -- Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 1:14 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job When running directly in an interactive allocation, I get the output below. Yes, Perlmutter is SLURM. Could there be some way to force the node-local MPICH to run without PMI? Using -launcher fork does not seem to have an effect. I can use a very minimal MPICH but I do need GPU support. Thanks $ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ./main.x host: x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov [mpiexec@x4217c5s3b0n0] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- ... mpiexec@x4217c5s3b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov:41641 --debug --rmk pbs --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Thursday, April 23, 2026 12:21 To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: Aurora: Trying to start MPICH inside bigger job /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x First, are you able to run hydra (MPICH's mpiexec) directly, i.e. ` /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ~/../main.x`? If that went well, then the key issue is we have two PMI environment concurrently active. One from PALs, assumably running PMIx; and one from hydra, assumably running PMI-1. This may confuse the MPI processes on which PMI protocol to use. What is the launcher on Perlmutter? Is it Slurm? It will have the same double active PMI environment issue, but I guess it happened to work. What is the error in MPI_Init are you seeing? Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 12:04 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: [mpich-devel] Aurora: Trying to start MPICH inside bigger job Hi I am trying to run a simulation ensemble where the system MPI coordinates many tasks, each of which is a call to a plain MPICH-built app (ExaEpi/AMReX). The app runs on its own, but when called from inside the bigger MPI job, it causes an error in MPI_Init(). This approach works on other systems like Perlmutter. I think I can reproduce this with something as simple as: /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x where main.x is a toy MPI program, mpich-git is my MPICH build on Aurora, and UNSETS is a variety of attempts to unset environment variables, none of this has worked so far. Is there any known reason why I am unable to run MPICH inside this context, when it does run on its own? Also, any other tips for debugging MPI startup might help. I am using the following but not getting much detail: export PMI_DEBUG=1 export PMIX_DEBUG=1 export HYDRA_DEBUG=1 I get: [mpiexec@x4217c5s7b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s7b0n0:37193 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS x4217c5s7b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 16 Thanks Justin -- Justin M Wozniak

1 0

Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job
by Wozniak, Justin M. 27 Apr '26

27 Apr '26

That worked, thanks! I just had to unset PMIX_NAMESPACE at runtime. -- Justin M Wozniak ________________________________ From: Zhou, Hui via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 13:44 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job I see the issue! The application main.x is linked to a version of MPICH build with PMIx and hydra only uses PMI1 or PMI2. You need rebuild MPICH without PMIx and relink the app with the custom built MPICH. There is a way to build MPICH with both PMIx and PMI1/2, but it is not well tested. Try specify both the configure option --with-pmix=... and --with-pmilib=mpich in building MPICH, and let me know if it works. -- Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 1:14 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job When running directly in an interactive allocation, I get the output below. Yes, Perlmutter is SLURM. Could there be some way to force the node-local MPICH to run without PMI? Using -launcher fork does not seem to have an effect. I can use a very minimal MPICH but I do need GPU support. Thanks $ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ./main.x host: x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov [mpiexec@x4217c5s3b0n0] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- ... mpiexec@x4217c5s3b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov:41641 --debug --rmk pbs --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Thursday, April 23, 2026 12:21 To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: Aurora: Trying to start MPICH inside bigger job /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x First, are you able to run hydra (MPICH's mpiexec) directly, i.e. ` /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ~/../main.x`? If that went well, then the key issue is we have two PMI environment concurrently active. One from PALs, assumably running PMIx; and one from hydra, assumably running PMI-1. This may confuse the MPI processes on which PMI protocol to use. What is the launcher on Perlmutter? Is it Slurm? It will have the same double active PMI environment issue, but I guess it happened to work. What is the error in MPI_Init are you seeing? Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 12:04 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: [mpich-devel] Aurora: Trying to start MPICH inside bigger job Hi I am trying to run a simulation ensemble where the system MPI coordinates many tasks, each of which is a call to a plain MPICH-built app (ExaEpi/AMReX). The app runs on its own, but when called from inside the bigger MPI job, it causes an error in MPI_Init(). This approach works on other systems like Perlmutter. I think I can reproduce this with something as simple as: /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x where main.x is a toy MPI program, mpich-git is my MPICH build on Aurora, and UNSETS is a variety of attempts to unset environment variables, none of this has worked so far. Is there any known reason why I am unable to run MPICH inside this context, when it does run on its own? Also, any other tips for debugging MPI startup might help. I am using the following but not getting much detail: export PMI_DEBUG=1 export PMIX_DEBUG=1 export HYDRA_DEBUG=1 I get: [mpiexec@x4217c5s7b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s7b0n0:37193 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS x4217c5s7b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 16 Thanks Justin -- Justin M Wozniak

1 0

Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job
by Zhou, Hui 23 Apr '26

23 Apr '26

I see the issue! The application main.x is linked to a version of MPICH build with PMIx and hydra only uses PMI1 or PMI2. You need rebuild MPICH without PMIx and relink the app with the custom built MPICH. There is a way to build MPICH with both PMIx and PMI1/2, but it is not well tested. Try specify both the configure option --with-pmix=... and --with-pmilib=mpich in building MPICH, and let me know if it works. -- Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 1:14 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job When running directly in an interactive allocation, I get the output below. Yes, Perlmutter is SLURM. Could there be some way to force the node-local MPICH to run without PMI? Using -launcher fork does not seem to have an effect. I can use a very minimal MPICH but I do need GPU support. Thanks $ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ./main.x host: x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov [mpiexec@x4217c5s3b0n0] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- ... mpiexec@x4217c5s3b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov:41641 --debug --rmk pbs --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Thursday, April 23, 2026 12:21 To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: Aurora: Trying to start MPICH inside bigger job /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x First, are you able to run hydra (MPICH's mpiexec) directly, i.e. ` /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ~/../main.x`? If that went well, then the key issue is we have two PMI environment concurrently active. One from PALs, assumably running PMIx; and one from hydra, assumably running PMI-1. This may confuse the MPI processes on which PMI protocol to use. What is the launcher on Perlmutter? Is it Slurm? It will have the same double active PMI environment issue, but I guess it happened to work. What is the error in MPI_Init are you seeing? Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 12:04 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: [mpich-devel] Aurora: Trying to start MPICH inside bigger job Hi I am trying to run a simulation ensemble where the system MPI coordinates many tasks, each of which is a call to a plain MPICH-built app (ExaEpi/AMReX). The app runs on its own, but when called from inside the bigger MPI job, it causes an error in MPI_Init(). This approach works on other systems like Perlmutter. I think I can reproduce this with something as simple as: /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x where main.x is a toy MPI program, mpich-git is my MPICH build on Aurora, and UNSETS is a variety of attempts to unset environment variables, none of this has worked so far. Is there any known reason why I am unable to run MPICH inside this context, when it does run on its own? Also, any other tips for debugging MPI startup might help. I am using the following but not getting much detail: export PMI_DEBUG=1 export PMIX_DEBUG=1 export HYDRA_DEBUG=1 I get: [mpiexec@x4217c5s7b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s7b0n0:37193 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS x4217c5s7b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 16 Thanks Justin -- Justin M Wozniak

1 0

Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job
by Wozniak, Justin M. 23 Apr '26

23 Apr '26

When running directly in an interactive allocation, I get the output below. Yes, Perlmutter is SLURM. Could there be some way to force the node-local MPICH to run without PMI? Using -launcher fork does not seem to have an effect. I can use a very minimal MPICH but I do need GPU support. Thanks $ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ./main.x host: x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov [mpiexec@x4217c5s3b0n0] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- ... mpiexec@x4217c5s3b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s3b0n0.hsn.cm.aurora.alcf.anl.gov:41641 --debug --rmk pbs --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. Abort(672810000): Fatal error in internal_Init: Internal MPI error!, error stack: internal_Init(70)...............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(204)...........: MPIR_pmi_init(225)..............: check_MPIR_CVAR_PMI_VERSION(158): Runtime environment uses unsupported PMI version PMI-1 or PMI-2. Aborting. [proxy:0@x4217c5s3b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Thursday, April 23, 2026 12:21 To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: Aurora: Trying to start MPICH inside bigger job /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x First, are you able to run hydra (MPICH's mpiexec) directly, i.e. ` /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ~/../main.x`? If that went well, then the key issue is we have two PMI environment concurrently active. One from PALs, assumably running PMIx; and one from hydra, assumably running PMI-1. This may confuse the MPI processes on which PMI protocol to use. What is the launcher on Perlmutter? Is it Slurm? It will have the same double active PMI environment issue, but I guess it happened to work. What is the error in MPI_Init are you seeing? Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 12:04 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: [mpich-devel] Aurora: Trying to start MPICH inside bigger job Hi I am trying to run a simulation ensemble where the system MPI coordinates many tasks, each of which is a call to a plain MPICH-built app (ExaEpi/AMReX). The app runs on its own, but when called from inside the bigger MPI job, it causes an error in MPI_Init(). This approach works on other systems like Perlmutter. I think I can reproduce this with something as simple as: /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x where main.x is a toy MPI program, mpich-git is my MPICH build on Aurora, and UNSETS is a variety of attempts to unset environment variables, none of this has worked so far. Is there any known reason why I am unable to run MPICH inside this context, when it does run on its own? Also, any other tips for debugging MPI startup might help. I am using the following but not getting much detail: export PMI_DEBUG=1 export PMIX_DEBUG=1 export HYDRA_DEBUG=1 I get: [mpiexec@x4217c5s7b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s7b0n0:37193 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS x4217c5s7b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 16 Thanks Justin -- Justin M Wozniak

1 0

Re: [mpich-devel] Aurora: Trying to start MPICH inside bigger job
by Zhou, Hui 23 Apr '26

23 Apr '26

/opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x First, are you able to run hydra (MPICH's mpiexec) directly, i.e. ` /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 ~/../main.x`? If that went well, then the key issue is we have two PMI environment concurrently active. One from PALs, assumably running PMIx; and one from hydra, assumably running PMI-1. This may confuse the MPI processes on which PMI protocol to use. What is the launcher on Perlmutter? Is it Slurm? It will have the same double active PMI environment issue, but I guess it happened to work. What is the error in MPI_Init are you seeing? Hui ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 23, 2026 12:04 PM To: Zhou, Hui via devel <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: [mpich-devel] Aurora: Trying to start MPICH inside bigger job Hi I am trying to run a simulation ensemble where the system MPI coordinates many tasks, each of which is a call to a plain MPICH-built app (ExaEpi/AMReX). The app runs on its own, but when called from inside the bigger MPI job, it causes an error in MPI_Init(). This approach works on other systems like Perlmutter. I think I can reproduce this with something as simple as: /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x where main.x is a toy MPI program, mpich-git is my MPICH build on Aurora, and UNSETS is a variety of attempts to unset environment variables, none of this has worked so far. Is there any known reason why I am unable to run MPICH inside this context, when it does run on its own? Also, any other tips for debugging MPI startup might help. I am using the following but not getting much detail: export PMI_DEBUG=1 export PMIX_DEBUG=1 export HYDRA_DEBUG=1 I get: [mpiexec@x4217c5s7b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s7b0n0:37193 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS x4217c5s7b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 16 Thanks Justin -- Justin M Wozniak

1 0

Aurora: Trying to start MPICH inside bigger job
by Wozniak, Justin M. 23 Apr '26

23 Apr '26

Hi I am trying to run a simulation ensemble where the system MPI coordinates many tasks, each of which is a call to a plain MPICH-built app (ExaEpi/AMReX). The app runs on its own, but when called from inside the bigger MPI job, it causes an error in MPI_Init(). This approach works on other systems like Perlmutter. I think I can reproduce this with something as simple as: /opt/cray/pals/1.8/bin/mpiexec \ env $UNSETS \ /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/mpiexec -n 2 \ ~/../main.x where main.x is a toy MPI program, mpich-git is my MPICH build on Aurora, and UNSETS is a variety of attempts to unset environment variables, none of this has worked so far. Is there any known reason why I am unable to run MPICH inside this context, when it does run on its own? Also, any other tips for debugging MPI startup might help. I am using the following but not getting much detail: export PMI_DEBUG=1 export PMIX_DEBUG=1 export HYDRA_DEBUG=1 I get: [mpiexec@x4217c5s7b0n0] Launch arguments: /lus/flare/projects/EpiCalib/sfw/mpich-git/bin/hydra_pmi_proxy --control-port x4217c5s7b0n0:37193 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_STDERR Abort(16): Fatal error in internal_Init: Internal MPI error! [proxy:0@x4217c5s7b0n0] Sending upstream hdr.cmd = CMD_EXIT_STATUS x4217c5s7b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 exited with code 16 Thanks Justin -- Justin M Wozniak

1 0

Re: [mpich-devel] MPICH with SYCL on Aurora
by Wozniak, Justin M. 23 Apr '26

23 Apr '26

This turned out to be an application-level thing, sorry. -- Justin M Wozniak ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, April 6, 2026 10:56 To: Zhou, Hui <zhouh(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Raffenetti, Ken <raffenet(a)anl.gov>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora Compared to the single process run. This is not the case when built with the system MPI. -- Justin M Wozniak ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Sunday, April 5, 2026 20:27 To: devel(a)mpich.org <devel(a)mpich.org>; Raffenetti, Ken <raffenet(a)anl.gov>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Hi Justin. 6x slower compared to which baseline? Hui Get Outlook for Android<https://urldefense.us/v3/__https://aka.ms/AAb9ysg__;!!G_uCfscf7eWS!dWLTYWC0…> ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Thursday, April 2, 2026 8:32:59 AM To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora That makes it run but is about 6x slower when running with 2 processes. Is there something else I can check? I get: # --with-device=ch4:ofi # *** Device Configuration # *** -------------------- # *** device : ch4:ofi # *** ipc feature : xpmem gpudirect # *** gpu support : ZE Note that I do not need any network support, as I am using this MPICH for node-local communication only. So I also do not need integrated process launch features. -- Justin M Wozniak ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Wednesday, April 1, 2026 9:10 To: Wozniak, Justin M. <woz(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov> Subject: Re: MPICH with SYCL on Aurora If building with Intel GPU support on Aurora, you should have --with-ze and --enable-ze-native=pvc in the configuration line. See https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/wiki/Using-MPIC…> for more details. Ken From: Wozniak, Justin M. <woz(a)anl.gov> Date: Tuesday, March 31, 2026 at 4:28 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org>, Harms, Kevin <harms(a)alcf.anl.gov> Subject: Re: MPICH with SYCL on Aurora I should have mentioned- I configured with --with-ze=no to avoid this build-time error: CC src/backend/ze/pup/yaksuri_zei_get_ptr_attr.lo OCLOC (spirv) src/backend/ze/pup/yaksuri_zei_pup_char.cl Could not determine device target: skl. Error: Cannot get HW Info for device skl. Invalid device error, trying to fallback to former ocloc libocloc_legacy1.so Couldn't load former ocloc libocloc_legacy1.so Command was: ocloc compile -file src/backend/ze/pup/yaksuri_zei_pup_char.cl -device skl -spv_only -out_dir src/backend/ze/pup -output_no_suffix -q -options "-I ./src/backend/ze/include -cl-std=CL2.0" make[2]: *** [Makefile:13671: src/backend/ze/pup/yaksuri_zei_pup_char.c] Error 223 make[2]: Leaving directory '/tmp/wozniak/mpich-5.0.0rc3-ze/src/mpi/datatype/typerep/yaksa' Maybe this is the main thing I need to address? Do I just need to force the ocloc device for Aurora? -- Justin M Wozniak ________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Tuesday, March 31, 2026 12:02 To: Raffenetti, Ken <raffenet(a)anl.gov>; devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora This is mpich-5.0.0rc3 , I will try that, thanks. -- Justin M Wozniak ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Tuesday, March 31, 2026 11:30 To: devel(a)mpich.org <devel(a)mpich.org>; Harms, Kevin <harms(a)alcf.anl.gov> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Which version of MPI is this? This might be a known issue in CMA support (fixed here https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7743__;!!G…>). You can try disabling CMA with MPIR_CVAR_CH4_CMA_ENABLE=0 to avoid that path or pull in the fix to your copy and rebuild. Ken From: Wozniak, Justin M. via devel <devel(a)mpich.org> Date: Tuesday, March 31, 2026 at 11:27 AM To: Harms, Kevin <harms(a)alcf.anl.gov>, devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: [mpich-devel] MPICH with SYCL on Aurora With MPIR_CVAR_REQUEST_ERR_FATAL=1 in a 2-process run, this looks like: Abort(270742287) on node 0: Fatal error in internal_Waitall: Other MPI error, error stack: internal_Waitall(126)..: MPI_Waitall(count=1, array_of_requests=0x797cdb0, array_of_statuses=0x7ca3fb0) failed MPIR_Waitall(916)......: MPIDI_IPC_rndv_cb(172).: MPIDI_CMA_copy_data(54): copy_iovs(202).........: process_vm_readv failed (errno 14) Abort(270742287) on node 1: Fatal error in internal_Waitall: Other MPI error, error stack: (same) This succeeds for 1-process with SYCL enabled or for 2-process with SYCL disabled in the app at configure time. The app looks like: $ ldd =agent libmpicxx.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpicxx.so.0 (0x0000146726aed000) libmpi.so.0 => /lus/flare/projects/EpiCalib/sfw/mpich-5.0.0rc3/lib/libmpi.so.0 (0x0000146725265000) libmkl_sycl_blas.so.5 => /opt/aurora/26.26.0/oneapi/mkl/latest/lib/libmkl_sycl_blas.so.5 (0x0000146721063000) (etc.) libstdc++.so.6 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libstdc++.so.6 (0x0000146708cd9000) libm.so.6 => /lib64/libm.so.6 (0x0000146708b77000) libgcc_s.so.1 => /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/gcc-13.4.0-hgnyg4p/lib64/libgcc_s.so.1 (0x0000146708b53000) libsycl.so.8 => /opt/aurora/26.26.0/oneapi/compiler/latest/lib/libsycl.so.8 (0x0000146708758000) libOpenCL.so.1 => /opt/aurora/26.26.0/support/libraries/khronos/default/lib64/libOpenCL.so.1 (0x0000146708743000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000014670871f000) libc.so.6 => /lib64/libc.so.6 (0x000014670852a000) libhwloc.so.15 => /opt/aurora/26.26.0/oneapi/tcm/latest/lib/libhwloc.so.15 (0x00001467082cc000) (etc.) Thanks -- Justin M Wozniak ________________________________ From: Harms, Kevin <harms(a)alcf.anl.gov> Sent: Monday, March 30, 2026 14:50 To: devel(a)mpich.org <devel(a)mpich.org> Cc: Wozniak, Justin M. <woz(a)anl.gov> Subject: Re: MPICH with SYCL on Aurora Justin, can you provide the specific error? kevin ________________________________________ From: Wozniak, Justin M. via devel <devel(a)mpich.org> Sent: Monday, March 30, 2026 2:24 PM To: devel(a)mpich.org Cc: Wozniak, Justin M. Subject: [mpich-devel] MPICH with SYCL on Aurora Hi I am trying to port a simulation ensemble workflow that runs ExaEpi/AMReX/SYCL to Aurora. The outer workflow uses the system MPI and we use MPICH to run the app with node-local parallelism using a hand-built MPICH. On Aurora, I get errors in early MPI calls that I think are due to SYCL. This approach works on NVIDIA systems like Perlmutter. Is there some simple way to make MPICH aware of SYCL? Thanks -- Justin M Wozniak

1 0

MPICH development call tomorrow
by Zhou, Hui 22 Apr '26

22 Apr '26

Dear MPICH family and friends, The next MPICH development tele-con will be tomorrow (Thursday) at 9am US Central time. agenda * Misc warning fix * Fix MPL_DBG_MSG_FMT * RMA window to share comm VCI * Check inject_size before fi_tinject * Collective Selection Redesign - update * Recently merged PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * Recently opened PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * New issues<https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues?q=is*3Ai… > To join the Meeting: https://urldefense.us/v3/__https://teams.microsoft.com/l/meetup-join/19*3am… If you would like a calendar invite, please send us a message. Thanks, -- Hui Zhou Principal Research Software Engineer Mathematics and Computer Science Argonne National Laboratory TEL: 630-252-3430

1 0

MPICH development call tomorrow
by Zhou, Hui 15 Apr '26

15 Apr '26

Dear MPICH family and friends, The next MPICH development tele-con will be tomorrow (Thursday) at 9am US Central time. agenda * MPICH v5.0.1 released * Collective Selection Redesign - update * Recently merged PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * Recently opened PRs<https://urldefense.us/v3/__https://github.com/pmodels/mpich/pulls?q=is*3Apr… > * New issues<https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues?q=is*3Ai… > To join the Meeting: https://urldefense.us/v3/__https://teams.microsoft.com/l/meetup-join/19*3am… If you would like a calendar invite, please send us a message. Thanks, -- Hui Zhou Principal Research Software Engineer Mathematics and Computer Science Argonne National Laboratory TEL: 630-252-3430

1 0

MPICH 5.0.1 released
by Raffenetti, Ken 10 Apr '26

10 Apr '26

A new stable release of MPICH, 5.0.1, is now available for download. This release addresses a number of user-reported issues. Thanks to all who contributed reports and fixes! For the full set of commits see https://urldefense.us/v3/__https://github.com/pmodels/mpich/compare/v5.0.0_… …v5.0.1/. You can find the release on our downloads page (https://urldefense.us/v3/__https://www.mpich.org/downloads/__;!!G_uCfscf7eW… ) Regards, The MPICH team =============================================================================== Changes in 5.0.1 =============================================================================== # Fix bad cast in release-gather collectives that caused data loss issues on Big-Endian 64b arches (s390x) # Fix issue with canceling MPI_ANY_SOURCE receive requests # Fix configuration issue when C++ compiler does not support complex types # Fix function signature issue in Hydra PBS support # Fix crash in MPI_Allreduce with MPI_LOGICAL type # Fix potential crash in multi-nic libfabric initialization # Fix memory leaks in Level Zero and PMIx support # Fix bug in CMA code when GPU support is enabled # Fix potential shared memory collision between multiple singleton processes on the same node # Add large count and other necessary aliases to ROMIO to avoid accidental profiling of internal MPI function usage # Add missing error checks in rndv and colletive composition code # Improve autogen.sh error message when autotools are too old

1 0