August 2023 - Ecp-aurora-sdk-discuss

Re: [Ecp-aurora-sdk-discuss] MPI executables hanging
by Bertoni, Colleen 17 Aug '23

17 Aug '23

Hi Jon, Nothing should have changed by default since August 9th (the last change in default was at the end of July.) Were you using the default SDK (2023.05.15.006) for everything, or possibly did you use a different SDK earlier? I would recommend trying to rebuild just to see if that works though (and save the current exe just in case we want to look at it more). If a newer build still fails, then it sounds like a bug we need to look into and/or report to Intel. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Rood, Jon via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, August 16, 2023 3:27 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] MPI executables hanging Hello, I have a job with an executable that was running successfully a few weeks ago (around August 9). Now I’m seeing the mpiexec call to my executable hanging indefinitely until walltime kills the job. I am able to get a CPU MPI executable to run with `export MPIR_CVAR_ENABLE_GPU=0`, but an mpiexec with a different executable with MPI ranks that run on the GPU still hangs no matter what I set MPIR_CVAR_ENABLE_GPU to. Has something changed recently? Should I rebuild my executable that was built on August 9 to account for any changes to the environment? Jon

1 0

short survey about MKL eigensolver usage (dsyevd)
by Bertoni, Colleen 16 Aug '23

16 Aug '23

Hi everyone, The MKL team asked us to ask application teams about use case for eigensolvers. Could you respond to me if: * You are using dsyevd from MKL on the GPU (or are interested in doing so) * If you are, are you running only on a single GPU / GPU tile? * Would you be interested in running a single node / multiple GPU eigensolve (this would allow for much larger inputs) * Would you be interested in a multi-node multi-GPU eigensolve in MKL? Thanks, Colleen

1 0

MPI executables hanging
by Rood, Jon 16 Aug '23

16 Aug '23

Hello, I have a job with an executable that was running successfully a few weeks ago (around August 9). Now I’m seeing the mpiexec call to my executable hanging indefinitely until walltime kills the job. I am able to get a CPU MPI executable to run with `export MPIR_CVAR_ENABLE_GPU=0`, but an mpiexec with a different executable with MPI ranks that run on the GPU still hangs no matter what I set MPIR_CVAR_ENABLE_GPU to. Has something changed recently? Should I rebuild my executable that was built on August 9 to account for any changes to the environment? Jon

1 0