November 2008 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Telecon ended early
by Richard Graham 26 Nov '08

26 Nov '08

Only two of showed up for the call ... Talk with you next week. Rich

1 0

[Mpi3-ft] my notes from 11/18/08 ft meeting at SC '08
by Howard Pritchard 25 Nov '08

25 Nov '08

Hello Folks, Here is a distillation of the notes I had from the meeting for mpi ft at SC'08. I wrote them more in the styles of minutes then high level concept notes. Please add or correct if you see problems/inaccuracies. In attendance at the meeting were Kannan Narashimhan, Thomas Herault, Greg Bronevetsky, Erez Hara, Hideyuki Jitsumoto, Greg Koenig, Rich Graham, and Howard Pritchard, and Josh Hursey. There was an attempt to organize the discussion around the most recent version of Rich's "Point-to-Point Communications recovery" document. Rich updated the document during the course of the meeting. The meeting began with a discussion of the "Purpose" section of this document. The document listed symmetric and asymmetric cases. There were questions about whether or not 'asymmetric' made sense here - how can it imply process death? Rich said the document was not intended to cover communication/network errors, but what to do when a process was dead/unreachable. It was decided to remove the mention of 'asymmetric' from this part of the document. At this point Rich brouight up that he would like to turn this into a paper suitable to submission to 2009 ACM International Conference on Computing Frontiers in about four weeks. There was general agreement with the "Errors" section of the document. Discussion next moved to the "Error States" section. The point was repeated that a application process will only be able to determine that MPI has detected an error when making an MPI call, using a communicator that is in an error state. There was more discussion about communicators within a process being in a 'consistent' state. Greg brought up the idea of a process as a group of "local communicators", with Kannan adding the idea of "global communicator" - a notion of local view vs global view. Greg made an analogy to memory consistency models. Erez wanted to understand better what it means to have an error state associated with a communicator - bringing up the notion of virtual connections. One of the problems discussed was whether to keep going if a communicator is being used for ranks that are still okay. Thomas advocates lazy notification. Erez pulled up the VC document and discussion continued about allowing a communicator to be used even if one or more 'vcs' was 'bad', as long as the application wasn't trying to use these 'vcs'. The problem of MPI_ANY_SOURCE was brought into the discussion. Questions were raised about what to do with received messages from a 'bad' vc which had not yet been matched. There were also questions as to whether or not this model was consistent. Thomas' drew up a diagram on a white board showing where he thought we may have a consistency issue - the problem requiring at least 3 bodies to show up. The conclusions at this point were that the app would get an error if - it tries to send to a rank /vc in a communicator which is in error - tries to receive from a rank/vc in a communicator which is in error - if it posts an MPI_ANY_SOURCE on a communicator with one or more bad vcs. For the last case, if the post does match a message, the data is returned along with the error indication. The app can at this point decide to repair the communicator or just keep going. The app can continue to use MPI_ANY_SOURCE, and, if messages are available to be received, MPI will continue to return message data and the error status. Its up to the application to know its not necessary to repair the communicator however. It can ignore the error return but may eventually hang. There was some dissatisfaction with this policy. Kannan suggested making it an option the application could set, presumably at application startup. Another idea considered would be to never hang in the MPI_ANY_SOURCE case but just return an error. Discussion returned to the 'consistent state' problem again - led by Greg and Kannan. The current thinking is two types of repair policies: repair with hole, and repair with replace. Erez brought up that the notion that the later only would work with MPI_COMM_WORLD -at least according to previous discussions. Rich said that isn't sufficient. This then led to a lengthy discussion about how to rebuild communicators in the case of the repair with replace policy. A scenario was outlined for the 'restarted/respawned' process: - original process dies/goes away - another process, detecting an error when using MPI with the 'errored' communicator, respawns the process as part of its communicator repair procedure - new process starts - out-of-band the restarted process (more accurately the MPI library internally) knows about communicators that existed and somehow repairs them The last step was the main point of further discussion. The issue of "naming" of communicators came up. There has to be some way for the restarted process to link the opaque communicator handles returned by mpi to the application. This "naming" convention would allow for reconstructing communicators. To avoid deadlock and address scalability, repair is considered to be non- blocking, if not local. A diagram of the discussion was drawn on the whiteboard at this point: respawning process respawned/new process -------------------------- -------------------------------- comm repair start comm rejoin/patch start comm repair complete comm rejoin/patch complete The discussion leaned toward having the rejoin/patching of the communicator be a local operation - or at least have the behavior of being local to the respawned process. There was also discussion of how to handle legacy libraries with a concept of "shutting down" communicators being raised. This was not discussed in detail. Erez brought up a final issue before the room had to be given up to the next group. Can the respawning process continue to use the communicator to exchange messages with "good" ranks while the communicator is being repaired? Howard

1 0

Re: [Mpi3-ft] Working meeting at SC
by Erez Haba 18 Nov '08

18 Nov '08

Four Seasons Hotel Austin 98 San Jacinto Boulevard Austin, Tx Phone: 512-478-4500 From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Monday, November 17, 2008 11:02 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Working meeting at SC Erez, Can you answer this one ? Rich On 11/18/08 12:04 AM, "Howard Pritchard" <howardp(a)cray.com> wrote: Hello Rich, To avoid confusion, which four season's hotel is this? There appear to be two four season's hotels near the convention center. Howard Richard Graham wrote: Re: [Mpi3-ft] Working meeting at SC Just a reminder that we are meeting tomorrow am. I have attached the latest version of the document I have been using to pull our discussions together. I have added more to it, following up with some users on open questions, and continuing to fill out missing details. Rich On 10/30/08 10:33 AM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: The meeting room is in the Four Seasons hotel, room #616. So far we have nine people that have expressed interest in participating. Looks like the room limit of 15 will not be a problem. Rich On 10/29/08 9:09 AM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: Based on the feedback from people, we will go ahead and meet on Tuesday, 11/18/2008, from 9am to 12pm. The exact location will be announced later on. Since space is limited (but should be more than enough to accommodate the on-going participants in the working group) please RSVP to me - we can have up to 15 people in the meeting room. Erez, Greg, and Josh - I already have you down in the head count. Thanks, Rich ________________________________ _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft ________________________________ _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft ________________________________ _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Working meeting at SC
by Howard Pritchard 17 Nov '08

17 Nov '08

1 0

Re: [Mpi3-ft] Working meeting at SC
by Richard Graham 17 Nov '08

17 Nov '08

Erez, Can you answer this one ? Rich On 11/18/08 12:04 AM, "Howard Pritchard" <howardp(a)cray.com> wrote: > Hello Rich, > > To avoid confusion, which four season's hotel is this? There appear to be two > four > season's hotels near the convention center. > > Howard > > Richard Graham wrote: >> Re: [Mpi3-ft] Working meeting at SC Just a reminder that we are meeting >> tomorrow am. I have attached the latest version of the document I have been >> using to pull our discussions together. I have added more to it, following >> up with some users on open questions, and continuing to fill out missing >> details. >> >> Rich >> >> >> On 10/30/08 10:33 AM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: >> >> >>> The meeting room is in the Four Seasons hotel, room #616. >>> >>> So far we have nine people that have expressed interest in participating. >>> Looks like the room limit of 15 will not be a problem. >>> >>> Rich >>> >>> >>> On 10/29/08 9:09 AM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: >>> >>> >>>> Based on the feedback from people, we will go ahead and meet on Tuesday, >>>> 11/18/2008, from 9am to 12pm. The exact location will be announced later >>>> on. Since space is limited (but should be more than enough to accommodate >>>> the on-going participants in the working group) please RSVP to me we can >>>> have up to 15 people in the meeting room. >>>> Erez, Greg, and Josh I already have you down in the head count. >>>> >>>> Thanks, >>>> Rich >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >

1 0

Re: [Mpi3-ft] Working meeting at SC
by Richard Graham 17 Nov '08

17 Nov '08

Just a reminder that we are meeting tomorrow am. I have attached the latest version of the document I have been using to pull our discussions together. I have added more to it, following up with some users on open questions, and continuing to fill out missing details. Rich On 10/30/08 10:33 AM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: > The meeting room is in the Four Seasons hotel, room #616. > > So far we have nine people that have expressed interest in participating. > Looks like the room limit of 15 will not be a problem. > > Rich > > > On 10/29/08 9:09 AM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: > >> Based on the feedback from people, we will go ahead and meet on Tuesday, >> 11/18/2008, from 9am to 12pm. The exact location will be announced later on. >> Since space is limited (but should be more than enough to accommodate the >> on-going participants in the working group) please RSVP to me we can have >> up to 15 people in the meeting room. >> Erez, Greg, and Josh I already have you down in the head count. >> >> Thanks, >> Rich >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] No con-call today
by Richard Graham 12 Nov '08

12 Nov '08

I am cancelling the regularly con-call scheduled for today, due to SC preparations. Just a reminder that we will have a meeting on Tuesday, 11/18/2008 9am noon. Rich

1 0

Re: [Mpi3-ft] Process failure document
by Hoppe, Hans-Christian 05 Nov '08

05 Nov '08

Folks, I've added some comments to the document, which you'll find appended to this message. Agree with Rich that getting the model for colective comms right has high priority, so it's OK to focus on this one. Questions I'd have to the group are - is our model symmetric - that is if 2n processes are split in the middle into two groups of n processes, will each group see the same errors and be able to recover? - supose processes A and B. A fails after a while, B notices and starts a repair with restore processes; in which state will a process A' be whrn it joins B after the repair? Will A' need to do any special repair call itself? Cheerio Hans-Christian Hoppe Principal Engineer Intel GmbH Phone: +49-2232-2090-11 Hermuelheimer Strasse 8a Fax: +49-2232-2090-29 50321 Bruehl, Germany ________________________________ From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Dienstag, 4. November 2008 22:34 To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Process failure document I have captured a lot of what we have discussed about process fault-tolerance, and filled in more missing gaps to help move us a long a bit faster in our discussions. Please take a look at the document before the call tomorrow. I would like to pick up discussing what to do when collective communications fail. There are still details missing that need to be added. No API's at this stage, just the "model". I ran this past 3 different application groups today - this seems to be along the lines of what they are looking for, and they had some very useful comments... Rich --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] Process failure document
by Thomas Herault 05 Nov '08

05 Nov '08

Le 4 nov. 08 ï¿½ 22:33, Richard Graham a ï¿½crit : > I have captured a lot of what we have discussed about process fault- > tolerance, and filled in more missing gaps to help move us a long a > bit faster in our discussions. Please take a look at the document > before the call tomorrow. I would like to pick up discussing what > to do when collective communications fail. There are still details > missing that need to be added. No APIï¿½s at this stage, just the > ï¿½modelï¿½. I ran this past 3 different application groups today ï¿½ > this seems to be along the lines of what they are looking for, and > they had some very useful comments... > > Rich > < > process_failure_tech > .txt>_______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft Hi, unfortunately, I cannot attend the conf call today. Here are a couple of comments about the attached file. About > - Failures that can be corrected by the MPI implementation > without input from the > application are not considered errors from MPI's perspective. > So a transient network > failure, or an link failure that can be routed around are not > considered failures. I agree with this. I think it also implies that the link-failure scenario A <-> B not OK, A <-> C OK, C <-> B OK, and A detects a failure of B, but not C cannot happen. My point is that if the application level is able to communicate information from A to B through C, then the library also can, and this link failure should not be notified as a process failure. Hence, process failures can originate from link failures only if there is a real network partition (except for the case of asymmetric failures). About asymmetric failures: do we have to consider this case? Does it happen? About Error States and communicators: > - A communicator that has lost a process will be defined to be in an > error state > - A process moves into error state, if it tries to interact with a > failed process There should be a concept of communicator, and communicator handle: this is what you refer as communicator / local part of the communicator in the last part of the file. Local parts of the communicators should be allowed to be in a more inconsistent state: the communicator handle enters the error state only when the process it belongs to tries to interact with a failed process belonging to the same communicator. About Error notification: - "synchronous" notification should also happen if A is waiting for the completion of some communication involving B (receive from B, or receive from ANY_SOURCE). In the case of MPI_ANY_SOURCE: the notification should not be guaranteed: if A receives from ANY_SOURCE, B fails and C sends a message to A, A should either enter the error state and notify the error, or notify the reception of a message from C. Both behaviors should be authorized. If no message arrive while A is in the receive, eventually, the error must be notified. - "asynchronous" notification: we still need to discuss this. For me, the standard should allow asynchronous notification to happen only when exiting MPI calls (but high quality implementations can trigger this at any time), but errors about any process on any communicator can be notified when exiting any MPI call. > > Error notification must be consistent, i.e. MPI must propagate a > single view of the system, > which may change over time. Is it possible, if we consider asymmetric failures cases? About recovery and Point-to-Point: > - caller specifies the mode of recovery - restore processes, > replace missing processes with > MPI_PROC_NULL, or eliminate the communicator. Eliminating the communicator should be some kind of collective. Which is contradictory with the locality of the correction. I agree that the recovery should be a local operation. Some of the recovery mechanisms should be collective (like eliminate the comm., maybe restore the process). Others should be local (replace with MPI_PROC_NULL). Two processes cannot decide differently on the recovery action for a same failure. About Communicator life cycle: > No > communications can process while the communicator is in > an error state (Question: > This could require checking error state all the time, > so may need to think of a > better way to do this). Maybe restrict this to communications involving the failed processes, or No communications can process while the local part of the communicator is in an error state. Bests, Thomas

1 0

[Mpi3-ft] Process failure document
by Richard Graham 04 Nov '08

04 Nov '08

I have captured a lot of what we have discussed about process fault-tolerance, and filled in more missing gaps to help move us a long a bit faster in our discussions. Please take a look at the document before the call tomorrow. I would like to pick up discussing what to do when collective communications fail. There are still details missing that need to be added. No API¹s at this stage, just the ³model². I ran this past 3 different application groups today this seems to be along the lines of what they are looking for, and they had some very useful comments... Rich

1 0