Unfortunately, the fact that “test3” is passing the “portname" on the command-line is a red-herring - connect/accept also fails when the “portname” is advertised by the job that opens the port using MPI_Publish_name and discovered by the other
job using MPI_Lookup_name. This is the test case that the sandbox code relies on. I also modified the “test3” example to use Publish/Lookup to figure out if there was any difference in the internal handling (inside Open MPI and/or inside PMIx) between these
situations. There is no difference in the final outcome or in the code path from dpm_connect_accept to the deadlock.
In
all cases, both processes get as far as ompi/dpm/dpm.c:398 (using git commit 5768336) and calls into PMIx_Connect. They both then get as far as opal/mca/pmix/pmix3x/pmix/src/client/pmix_client_connect.c:102 (same git hash), i.e.
PMIX_WAIT_THREAD(&cb->lock);
What happens next is an infinite loop that (at least) includes a whole bunch of calls to "HASH:STORE rank -2 key pmix.<various>” output messages that burns a couple of CPUs at 100% and shifts data on the local network at max-bandwidth.
The MacOS Activity Monitor shows two orterun processes at 50% CPU each and one orte-server process at 100%. Network usage statistics are roughly in the same proportion.
It’s going to be hard to progress any further without knowing a starting point for the code doing the HASH:STORE operations. I guess it’s the PMIx progress threads trying to complete the PMIX_PTL_SEND_RECV operations pushed onto the event queues in PMIx_Connect_nb
but that isn’t helping me all that much.
Cheers,
Dan.
—
Dr Daniel Holmes PhD
Applications Consultant in HPC Research
d.holmes@epcc.ed.ac.uk
Phone: +44 (0) 131 651 3465
Mobile: +44 (0) 7940 524 088
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD
—
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
—