July 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Telecon this week
by Richard Graham 30 Jul '08

30 Jul '08

Thanks for catching this - you are correct. Rich On 7/30/08 12:48 PM, "Martin Schulz" <schulzm(a)llnl.gov> wrote: > Rich, > > At 08:26 AM 7/30/2008, Richard Graham wrote: >> The call this week will be: >> July 18, 2008 - 12-2 pm EDT, 5-7 pm GMT, US Toll Free number: >> 877-801-8130, Toll number: 1-203-692-8690, Code: 1044056 > > I assume you mean August 1, 2008 and the rest of the information > stays the same? > > Thanks, > > Martin > > > >> I expect that we will talk about the impact of failure via use-case >> scenarios and via the APIs, as we agreed to on the last call. >> >> Rich >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Telecon this week
by Martin Schulz 30 Jul '08

30 Jul '08

Rich, At 08:26 AM 7/30/2008, Richard Graham wrote: >The call this week will be: > July 18, 2008 - 12-2 pm EDT, 5-7 pm GMT, US Toll Free number: >877-801-8130, Toll number: 1-203-692-8690, Code: 1044056 I assume you mean August 1, 2008 and the rest of the information stays the same? Thanks, Martin >I expect that we will talk about the impact of failure via use-case >scenarios and via the APIs, as we agreed to on the last call. > >Rich > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-ft] Network failure use-cases
by Greg Bronevetsky 30 Jul '08

30 Jul '08

Here's are the use-cases for network failures. I've attached the original .doc document, which has better formatting. The idea here is to define several types of network failures and the types of notification behaviors that applications might want from MPI. For each situation I list several options. We'll need to work out which option we actually want or whether we want to support multiple options and allow applications to choose the one it needs. The notification API currently on the Wiki would be able to support all of the options below. I. Network partition Description of event: a high level switch or major cable fails, causing a portion of the compute nodes to be inaccessible from the rest of the nodes for an unknown period of time. Desired notification policy options: Notification of failure 1. All processes are preemptively notified of failure before they try to communicate with processes in another partition. 2. Process only notified when it tries to communicate with node in another partition. Notification of repair 1. Application never notified of repair that fixes the network partition. Application responsible for ensuring that different partitions don't interfere with each other. 2. All processes are notified when partition is repaired. 3. Application must explicitly designate a process to be informed of the repair and is only notified if it has done so. II. Link Failure Description of event: The failure of a network link or low-level switch prevents a subset of processes from communicating with another subset. For example, in a statically-routed torus topology the failure of one or more key links or nodes can prevent a specific pair of nodes from communicating while leaving other nodes free to communicate with each other. Desired notification policy options: Notification of failure (processes A and B can't communicate with each other) 1. Processes A and B preemptively notified of failure before they try to communicate with each other. 2. When process A tries to communicate with B or vice versa, it is informed of failure. 3. When process C tries to perform a collective operation that involves both A and B, it is informed of the failure. Notification of repair 1. Both A and B notified when connection between them is repaired. 2. Application must explicitly designate a process to be informed of the repair and is only notified if it has done so. III. Global or local network performance degradation Description of event: performance is degraded because * (global) Some replicated high-level switches have failed causing all cross-machine communication to pass through fewer switches. * (global) The network is being shared with another application, which has entered a communication-heavy phase. * (local) Some replicated low-level switches have failed causing all local communication to pass through fewer switches. * (local) A given network wire has become frayed, causing its re-transmit rate to increase significantly. Desired notification policy options: Notification of degradation 1. All affected processes are informed before their next communication operation. 2. Each affected process informed at the time of its next communication operation. 3. Each process that participates in a collective call with an affected process is informed at the time of the call. 4. Application must explicitly designate a process to be informed of the degradation and is only notified if it has done so. Notification of repair 1. All affected processes are informed before their next communication operation. 2. Each affected process informed at the time of its next communication operation. 3. Application must explicitly designate a process to be informed of the repair and is only notified if it has done so. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

[Mpi3-ft] FW: MPI 3.0 FT WG: Error notification
by Erez Haba 30 Jul '08

30 Jul '08

FYI, I did not post this to the wiki pages yet. -----Original Message----- From: Erez Haba Sent: Tuesday, July 29, 2008 11:20 AM To: 'Narasimhan, Kannan' Subject: RE: MPI 3.0 FT WG: Error notification Hi Kannan, I didn't plan to define the error semantics for the collectives (although we should). At first I think I'd focus on defining the error semantics as it is today, to define when an error should be returned (and error handler should be called). The purpose is to better define error handling for libraries, to allow them to recover on their own turf. The idea is to define the error semantic in a way that is compatible with today's programs. Here are the few rules I have in mind, 1. errors are returned per call site, and associated with the call context 2. introduce the concept of a virtual connection 3. a vc's can move to an error state, once unable to communicate with its peer 3. a vc move to an error state cause all the communicators associated with it move to an error state 4. a vc error state affects only calls associated with that vc 5. communicator error state affects only receives with MPI_ANY_SOURCE Examples, Example 1: (running on rank 0) MPI_Irecv(source=4, &request4) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // the error is associated with request4 and thus returned in the MPI_Wait call // MPI_Wait(request4) Example 2: (running on rank 0) MPI_Irecv(source=any, &request) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // The communicator set to the error state, any receives with MPI_ANY_SOURCE // are set to error state and thus return an error in their wait call // MPI_Wait(request) Example 3: (running on rank 0) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // The communicator set to the error state, any receives with MPI_ANY_SOURCE // are set to error state and thus return an error. // MPI_Recv(source=any) // // VC 4 is set to the error state, any usage of that VC will return an error. // MPI_Recv(source=4) MPI_Send(dest=4) Here, continue with examples using multiple communicators. Make sense? Let me know what you think, I'll post that (and more, to the FT wiki pages) Thanks, .Erez -----Original Message----- From: Narasimhan, Kannan [mailto:[email protected]] Sent: Tuesday, July 29, 2008 4:04 AM To: Erez Haba Cc: Narasimhan, Kannan Subject: MPI 3.0 FT WG: Error notification Hi Erez, Following up on our discussion in the MPI 3.0 FT Working group meeting --- Did U get to start on the error semantics/notification for a sample MPI collective call? Let me know if you want to bounce some options/ideas on this topic -- we can hash it out over email/phone call before this Friday's WG confcall. Thanx! Kannan

1 0

[Mpi3-ft] Telecon this week
by Richard Graham 30 Jul '08

30 Jul '08

The call this week will be: July 18, 2008 - 12-2 pm EDT, 5-7 pm GMT, US Toll Free number: 877-801-8130, Toll number: 1-203-692-8690, Code: 1044056 I expect that we will talk about the impact of failure via use-case scenarios and via the APIs, as we agreed to on the last call. Rich

1 0

Re: [Mpi3-ft] Suggestion and self introduction
by Hideyuki Jitsumoto 21 Jul '08

21 Jul '08

Richerd, Yes, it's OK. Then, I updated my comment file, so please re-download it. Thank you, Hideyuki Jitsumoto. On Sun, Jul 20, 2008 at 9:28 PM, Richard Graham <rlgraham(a)ornl.gov> wrote: > Hideyuki, > Is it ok if we post this on the ft working group pages, so people can > easily view this with the rest of the document ? > > Thanks, > Rich > > > On 7/18/08 3:59 PM, "Hideyuki Jitsumoto" <jitumoto(a)smg.is.titech.ac.jp> > wrote: > >> MPI Fault Tolerance Group Member, >> >> Thank you for answering my 'long' question today. >> I'm a researcher on Tokyo Institute of Technology and research about >> MPI's fault tolerant. >> >> Today is my first participation in this meeting, so I send my >> suggestion(stance?). >> http://matsu-www.is.titech.ac.jp/~jitumoto/ftcomment_HideyukiJ.pdf >> >> I'm sorry if my idea misses the point. >> I consider that I want to take an active part in this meeting. >> >> -- >> Sincerely Yours, >> Hideyuki Jitsumoto (jitumoto(a)matsulab.is.titech.ac.jp) >> Tokyo Institute of Technology >> Global Scientific Information and Computing center (Matsuoka Lab.) >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Sincerely Yours, Hideyuki Jitsumoto (jitumoto(a)matsulab.is.titech.ac.jp) Tokyo Institute of Technology Global Scientific Information and Computing center (Matsuoka Lab.)

1 0

Re: [Mpi3-ft] Suggestion and self introduction
by Richard Graham 20 Jul '08

20 Jul '08

Hideyuki, Is it ok if we post this on the ft working group pages, so people can easily view this with the rest of the document ? Thanks, Rich On 7/18/08 3:59 PM, "Hideyuki Jitsumoto" <jitumoto(a)smg.is.titech.ac.jp> wrote: > MPI Fault Tolerance Group Member, > > Thank you for answering my 'long' question today. > I'm a researcher on Tokyo Institute of Technology and research about > MPI's fault tolerant. > > Today is my first participation in this meeting, so I send my > suggestion(stance?). > http://matsu-www.is.titech.ac.jp/~jitumoto/ftcomment_HideyukiJ.pdf > > I'm sorry if my idea misses the point. > I consider that I want to take an active part in this meeting. > > -- > Sincerely Yours, > Hideyuki Jitsumoto (jitumoto(a)matsulab.is.titech.ac.jp) > Tokyo Institute of Technology > Global Scientific Information and Computing center (Matsuoka Lab.) > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Suggestion and self introduction
by Hideyuki Jitsumoto 18 Jul '08

18 Jul '08

MPI Fault Tolerance Group Member, Thank you for answering my 'long' question today. I'm a researcher on Tokyo Institute of Technology and research about MPI's fault tolerant. Today is my first participation in this meeting, so I send my suggestion(stance?). http://matsu-www.is.titech.ac.jp/~jitumoto/ftcomment_HideyukiJ.pdf I'm sorry if my idea misses the point. I consider that I want to take an active part in this meeting. -- Sincerely Yours, Hideyuki Jitsumoto (jitumoto(a)matsulab.is.titech.ac.jp) Tokyo Institute of Technology Global Scientific Information and Computing center (Matsuoka Lab.)

1 0

Re: [Mpi3-ft] Telecon this week
by Joseph Ruscio 18 Jul '08

18 Jul '08

On Jul 18, 2008, at 9:11 AM, Narasimhan, Kannan wrote: > Are we on for today -- waiting on a music loop ... > > Thanx! > Kannan Mike and I were on as well, we just got timed out by the system ... -Joe

1 0

Re: [Mpi3-ft] Telecon this week
by Richard Graham 18 Jul '08

18 Jul '08

I called the wrong number, so if you want, please try calling again. Rich On 7/17/08 9:33 AM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: > The call this week will be: > July 18, 2008 - 12-2 pm EDT, 5-7 pm GMT, US Toll Free number: > 877-801-8130, Toll number: 1-203-692-8690, Code: 1044056 > > Also, since we now have participation in this working group from Asia, > Europe, and the US, we need to come up with a schedule that allows all to > participate at some level. Several ideas come to mind: > - Alternate meeting times. For example 12 EDT and 6 or 7 EDT > - Have a call early in the morning EDT, say 7 or 8 EDT > > Any other suggestions ? Any other preferences ? > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0