August 2010 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Teleconf & Status Update
by Joshua Hursey 19 Aug '10

19 Aug '10

As some of you may know I recently finished my Ph.D. at Indiana University and am now a postdoc at ORNL. I am working with Rich Graham and Ricky Kendall on application fault readiness at scale with a focus on MPI development. So I will still be active in the working group (possibly even more so). Rich asked me to lead the teleconfs for the working group for a while. Our next teleconf is scheduled for Aug. 25 at Noon Eastern. I'll send out the contact information the day before the call. I updated the wiki with the teleconf schedule at bottom. What active proposals are folks working on? Or, worded another way, are there topics that people would like to discuss on the teleconf next week? The topics I know of: - Piggybacking: Scheduled to report back to the forum at the next MPI Forum meeting on 9/16. - Non-blocking dynamics: Proposal is finished, implementation is needed. - Run-though process failure proposal: I'll have some more details on this for the call next week. - Others? Cheers, Josh Remaining Teleconf/Meeting Schedule for 2010: --------------------------------------------- 8/25 - Teleconf 9/8 - Teleconf 9/16 - MPI Forum WG Meeting [CA] (Piggybacking Plenary) 9/29 - Teleconf 10/13 - MPI Forum WG Meeting [Chicago] 10/27 - Teleconf 11/10 - Teleconf 11/24 - Teleconf (tentative) 12/8 - MPI Forum WG Meeting [CA] 12/22 - Teleconf (tentative)

1 0

Re: [Mpi3-ft] Question about MPI_ANY_SOURCE and process failures
by Graham, Richard L. 19 Aug '10

19 Aug '10

I tend to agree - that "all proces" option does not make sense. Rich On 8/19/10 3:09 PM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: I think it should be if ANY process fails on the communicator then it should return an error. I don't think it should be ALL, per your comments below. Additionally, ALL is not workable even if we do {ALL-self} since the intention of the receive is to receive one message, and the MPI interface cannot assume that the remaining processes will ever provide a message to the application. NEVER is probably not what we want either for the same reason. If only a subset of the processes in the communicator will ever send a message (due to app. design) the MPI library does not know if the procs that have failed are important or not. So I think the ANY process fail option is the only one that makes sense here. The state of the 'status' object should point to one of the process failures. If there are concurrent failures it is hard to tell which was first, so the semantics should probably just say that any one of the failed processes will be identified. Then the user should use a 'validate' command to figure out which ones have failed. -- Josh P.S. I am starting to work on a slightly more formal fail-though proposal for the group. This separates the interface/semantic issues for fail-through from recovery. This will help get us through some of the broader issues of stability before complicating the discussion with (multiple, concurrent) recoveries. More on this in the next couple weeks. On Aug 19, 2010, at 2:46 PM, Solt, David George wrote: > err = MPI_Recv(....., rankX, ..., comm, status); > > if communication to rankX fails, this receive will return with err. > > Err = MPI_Recv(...., MPI_ANY_SOURCE, ..., comm, status); > > When does this MPI_Recv return a failure? When ANY rank in comm is unreachable or when ALL ranks in comm are unreachable. Since self is always reachable, the ALL option is really NEVER. > > We had been assuming ALL/NEVER but will likely change to ANY. In such case, status points to the first failed rank that could have matched the request. > > Thanks, > Dave > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Question about MPI_ANY_SOURCE and process failures
by Joshua Hursey 19 Aug '10

19 Aug '10

I think it should be if ANY process fails on the communicator then it should return an error. I don't think it should be ALL, per your comments below. Additionally, ALL is not workable even if we do {ALL-self} since the intention of the receive is to receive one message, and the MPI interface cannot assume that the remaining processes will ever provide a message to the application. NEVER is probably not what we want either for the same reason. If only a subset of the processes in the communicator will ever send a message (due to app. design) the MPI library does not know if the procs that have failed are important or not. So I think the ANY process fail option is the only one that makes sense here. The state of the 'status' object should point to one of the process failures. If there are concurrent failures it is hard to tell which was first, so the semantics should probably just say that any one of the failed processes will be identified. Then the user should use a 'validate' command to figure out which ones have failed. -- Josh P.S. I am starting to work on a slightly more formal fail-though proposal for the group. This separates the interface/semantic issues for fail-through from recovery. This will help get us through some of the broader issues of stability before complicating the discussion with (multiple, concurrent) recoveries. More on this in the next couple weeks. On Aug 19, 2010, at 2:46 PM, Solt, David George wrote: > err = MPI_Recv(....., rankX, ..., comm, status); > > if communication to rankX fails, this receive will return with err. > > Err = MPI_Recv(...., MPI_ANY_SOURCE, ..., comm, status); > > When does this MPI_Recv return a failure? When ANY rank in comm is unreachable or when ALL ranks in comm are unreachable. Since self is always reachable, the ALL option is really NEVER. > > We had been assuming ALL/NEVER but will likely change to ANY. In such case, status points to the first failed rank that could have matched the request. > > Thanks, > Dave > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Question about MPI_ANY_SOURCE and process failures
by Bronevetsky, Greg 19 Aug '10

19 Aug '10

I believe that we agreed on the ANY option. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Solt, David George > Sent: Thursday, August 19, 2010 11:47 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] Question about MPI_ANY_SOURCE and process failures > > err = MPI_Recv(....., rankX, ..., comm, status); > > if communication to rankX fails, this receive will return with err. > > Err = MPI_Recv(...., MPI_ANY_SOURCE, ..., comm, status); > > When does this MPI_Recv return a failure? When ANY rank in comm is > unreachable or when ALL ranks in comm are unreachable. Since self is > always reachable, the ALL option is really NEVER. > > We had been assuming ALL/NEVER but will likely change to ANY. In such > case, status points to the first failed rank that could have matched the > request. > > Thanks, > Dave > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Question about MPI_ANY_SOURCE and process failures
by Solt, David George 19 Aug '10

19 Aug '10

err = MPI_Recv(....., rankX, ..., comm, status); if communication to rankX fails, this receive will return with err. Err = MPI_Recv(...., MPI_ANY_SOURCE, ..., comm, status); When does this MPI_Recv return a failure? When ANY rank in comm is unreachable or when ALL ranks in comm are unreachable. Since self is always reachable, the ALL option is really NEVER. We had been assuming ALL/NEVER but will likely change to ANY. In such case, status points to the first failed rank that could have matched the request. Thanks, Dave

1 0