August 2008 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Cancel tomorrow's con call
by Richard Graham 28 Aug '08

28 Aug '08

With the MPI Forum meeting next week I have too much preparation work, so I am cancelling the call scheduled for tomorrow, 8/29/2008. We do have a three hour face-to-face session scheduled next week. Rich

1 0

Re: [Mpi3-ft] Call this week (FT use cases)
by Erez Haba 21 Aug '08

21 Aug '08

Hi Rich, Thank you for putting this document together. Here are few of my thoughts; Prioritization: I think that we should prioritize these use cases as to what we think it is more important (actually what our customers think is more important) to enable us better focus on the right problems. My prioritization is as follow (extracted from your list), 1. Process failure 1.1 single process failure One process failed; either crashed or complete lack of contact from any of the connected ranks. 1.2 multiple process failure Multiple processes failed; either crashed or complete lack of contact to any of them from the connected ranks. 2. Link failure 2.1 two ranks/nodes cannot communicate directly, while other can communicate with both of them 2.2 group of nodes cannot communicate while other nodes can communicate with both groups 3. network partition 3.1 network failure partitioning the ranks to two or more disconnected groups (each group is connected) (there is some similarity here to process failure as seen from one group view) 4. performance degradation 4.1 the throughput, the latency or processing time has degraded between two ranks 4.2 the overall performance has degraded across all or most ranks (e.g., network is flooded; all processes moved to power save state) I think that we can discuss these scenarios and consolidate some of them for the sake of simplicity. However I think that they require discussion as the application might want to behave differently. Expected Application Action: I also found it to be a big leap going from the network case to the notification mechanism. In the use case document I would expect to understand first what are the possible actions that the application might want to take. Having that will give us some insight as to what are the possible steps required to enable the application FT. Layered approach: I think that when coming to enable FT for MPI application I would like to go with some layered approach where in each level we enable more FT (but with an associate cost) this allow us to better think about how to enable FT and for applications to choose their level of FT and cost. For example, we can layer our approach for solution(s) as follow 1. Error Reporting Rules This can actually go into mpi unrelated to FT. This enables applications to handle errors at the call site with the right context (rather than in inappropriate call site). This also enables libraries and framework to handle their own errors rather than other library errors. 2. Collectives/Communication integrity validation Enable applications to check that all communication on a specific communicator has been successful across all ranks participating. This enables all ranks to pick the right code path (error/success) once the collective communication is complete. 3. Error notification Enable applications to get notification if something is wrong in their world. This enables the scenario where applications go for a very long computation and would like to know during the computation if any of the other ranks died so it can be replaced. 4. Repairing Once an error was detected provide as set of API's to enable repairing the situation; either by spawning a replacement process; that are part of the existing communicators; or by providing a mechanism to control a subset of processes. (we haven't discuss this much, but in our short discussion we already found a set of issues; like restoring the state to re-spawned process) I think that we also need to discuss that our approach is application assisted FT rather than automatically provide an MPI FT approach. Makes sense to you? Thanks, .Erez From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Wednesday, August 20, 2008 6:22 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Call this week Just a reminder that we have our weekly call scheduled this Friday, 8/22/2008. Friday - 12-2 pm EDT, 5-7 pm GMT US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 The plan is to present our proposed approach for error notification in MPI-3. I have combined several different use case scenarios from various documents, as well as a high level description of Greg's proposal for event notification. We should go over this document on the call Friday - I don't think it is self-consistent as it reads, but would rather have us talk about this before making changes. We have nothing about MPI-I/O errors, and need to consider these too. Rich

1 0

[Mpi3-ft] Call this week
by Richard Graham 20 Aug '08

20 Aug '08

Just a reminder that we have our weekly call scheduled this Friday, 8/22/2008. Friday - 12-2 pm EDT, 5-7 pm GMT US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 The plan is to present our proposed approach for error notification in MPI-3. I have combined several different use case scenarios from various documents, as well as a high level description of Greg¹s proposal for event notification. We should go over this document on the call Friday I don¹t think it is self-consistent as it reads, but would rather have us talk about this before making changes. We have nothing about MPI-I/O errors, and need to consider these too. Rich

1 0

[Mpi3-ft] Call today
by Richard Graham 15 Aug '08

15 Aug '08

Just a reminder that we have a call scheduled today. I expect it will be short, as most of what we need to do is get a document ready by the mid the week of 8/25. Friday - 12-2 pm EDT, 5-7 pm GMT US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 Rich

1 0

Re: [Mpi3-ft] Regular weekly call
by Adam Moody 08 Aug '08

08 Aug '08

Hi Rich, I know the teleconference service which Torsten uses at Indiana for the Collectives WG records the conversations so that someone can listen to the recording in case they missed the call. Something like this may be helpful for this group as well since there is widespread international participation in the group, which makes it hard to schedule a time slot for the call. -Adam Richard Graham wrote: >In our last telecon we agreed to change the meeting frequency to once a >week, to help move us forward at a faster rate. We will keep the meeting at >the same time, rather than move it forward by an hour, as it accommodates a >greater number of people. We will not meet the week of our face-to-face >Forum meetings. Here are call details: > > Friday - 12-2 pm EDT, 5-7 pm GMT > US Toll Free number: 877-801-8130 > Toll number: 1-203-692-8690 > Access Code: 1044056 > >Rich > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > >

1 0

Re: [Mpi3-ft] Regular weekly call
by Hideyuki Jitsumoto 08 Aug '08

08 Aug '08

Everyone, Today (and probably next week), I stay in the countryside of JAPAN. So, very sorry, i can't get your talking clearly. And now, my communication-status is goes more and more unclear... then, today, I goes sleep sorry. Thank you, Hideyuki Jitsumoto On Mon, Aug 4, 2008 at 6:54 AM, Richard Graham <rlgraham(a)ornl.gov> wrote: > In our last telecon we agreed to change the meeting frequency to once a > week, to help move us forward at a faster rate. We will keep the meeting at > the same time, rather than move it forward by an hour, as it accommodates a > greater number of people. We will not meet the week of our face-to-face > Forum meetings. Here are call details: > > Friday - 12-2 pm EDT, 5-7 pm GMT > US Toll Free number: 877-801-8130 > Toll number: 1-203-692-8690 > Access Code: 1044056 > > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Sincerely Yours, Hideyuki Jitsumoto (jitumoto(a)matsulab.is.titech.ac.jp) Tokyo Institute of Technology Global Scientific Information and Computing center (Matsuoka Lab.)

1 0

[Mpi3-ft] Process failure use case
by Richard Graham 08 Aug '08

08 Aug '08

Here is the process failure use case. All the usual caveats this is a starting point. Rich

1 0

Re: [Mpi3-ft] MPI 3.0 FT WG: Error notification
by Erez Haba 08 Aug '08

08 Aug '08

I put the two suggestions in this paper on the FT wiki pages; as, https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/error_report_rules https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/comm_integrity linked from the FT page https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/FaultToleranceWikiPage thanks, .Erez -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Erez Haba Sent: Thursday, August 07, 2008 6:51 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] MPI 3.0 FT WG: Error notification Hi all, Attached is an update on the suggestion below. This is still a very early draft where I try to capture some of the ideas and discussions we're having. I expect to add more content to this document as we understand it better. I think that this is still very very basic for FT and we have still a long way to go. Thanks, .Erez -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Erez Haba Sent: Wednesday, July 30, 2008 9:27 AM To: MPI 3.0 Fault Tolerance Subject: [Mpi3-ft] FW: MPI 3.0 FT WG: Error notification FYI, I did not post this to the wiki pages yet. -----Original Message----- From: Erez Haba Sent: Tuesday, July 29, 2008 11:20 AM To: 'Narasimhan, Kannan' Subject: RE: MPI 3.0 FT WG: Error notification Hi Kannan, I didn't plan to define the error semantics for the collectives (although we should). At first I think I'd focus on defining the error semantics as it is today, to define when an error should be returned (and error handler should be called). The purpose is to better define error handling for libraries, to allow them to recover on their own turf. The idea is to define the error semantic in a way that is compatible with today's programs. Here are the few rules I have in mind, 1. errors are returned per call site, and associated with the call context 2. introduce the concept of a virtual connection 3. a vc's can move to an error state, once unable to communicate with its peer 3. a vc move to an error state cause all the communicators associated with it move to an error state 4. a vc error state affects only calls associated with that vc 5. communicator error state affects only receives with MPI_ANY_SOURCE Examples, Example 1: (running on rank 0) MPI_Irecv(source=4, &request4) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // the error is associated with request4 and thus returned in the MPI_Wait call // MPI_Wait(request4) Example 2: (running on rank 0) MPI_Irecv(source=any, &request) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // The communicator set to the error state, any receives with MPI_ANY_SOURCE // are set to error state and thus return an error in their wait call // MPI_Wait(request) Example 3: (running on rank 0) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // The communicator set to the error state, any receives with MPI_ANY_SOURCE // are set to error state and thus return an error. // MPI_Recv(source=any) // // VC 4 is set to the error state, any usage of that VC will return an error. // MPI_Recv(source=4) MPI_Send(dest=4) Here, continue with examples using multiple communicators. Make sense? Let me know what you think, I'll post that (and more, to the FT wiki pages) Thanks, .Erez -----Original Message----- From: Narasimhan, Kannan [mailto:[email protected]] Sent: Tuesday, July 29, 2008 4:04 AM To: Erez Haba Cc: Narasimhan, Kannan Subject: MPI 3.0 FT WG: Error notification Hi Erez, Following up on our discussion in the MPI 3.0 FT Working group meeting --- Did U get to start on the error semantics/notification for a sample MPI collective call? Let me know if you want to bounce some options/ideas on this topic -- we can hash it out over email/phone call before this Friday's WG confcall. Thanx! Kannan _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] MPI 3.0 FT WG: Error notification
by Erez Haba 07 Aug '08

07 Aug '08

Hi all, Attached is an update on the suggestion below. This is still a very early draft where I try to capture some of the ideas and discussions we're having. I expect to add more content to this document as we understand it better. I think that this is still very very basic for FT and we have still a long way to go. Thanks, .Erez -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Erez Haba Sent: Wednesday, July 30, 2008 9:27 AM To: MPI 3.0 Fault Tolerance Subject: [Mpi3-ft] FW: MPI 3.0 FT WG: Error notification FYI, I did not post this to the wiki pages yet. -----Original Message----- From: Erez Haba Sent: Tuesday, July 29, 2008 11:20 AM To: 'Narasimhan, Kannan' Subject: RE: MPI 3.0 FT WG: Error notification Hi Kannan, I didn't plan to define the error semantics for the collectives (although we should). At first I think I'd focus on defining the error semantics as it is today, to define when an error should be returned (and error handler should be called). The purpose is to better define error handling for libraries, to allow them to recover on their own turf. The idea is to define the error semantic in a way that is compatible with today's programs. Here are the few rules I have in mind, 1. errors are returned per call site, and associated with the call context 2. introduce the concept of a virtual connection 3. a vc's can move to an error state, once unable to communicate with its peer 3. a vc move to an error state cause all the communicators associated with it move to an error state 4. a vc error state affects only calls associated with that vc 5. communicator error state affects only receives with MPI_ANY_SOURCE Examples, Example 1: (running on rank 0) MPI_Irecv(source=4, &request4) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // the error is associated with request4 and thus returned in the MPI_Wait call // MPI_Wait(request4) Example 2: (running on rank 0) MPI_Irecv(source=any, &request) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // The communicator set to the error state, any receives with MPI_ANY_SOURCE // are set to error state and thus return an error in their wait call // MPI_Wait(request) Example 3: (running on rank 0) // // error in the communication with rank4 detected, the communicator moves to // an error state, but the send call returns success // MPI_Send(dest=3) // // The communicator set to the error state, any receives with MPI_ANY_SOURCE // are set to error state and thus return an error. // MPI_Recv(source=any) // // VC 4 is set to the error state, any usage of that VC will return an error. // MPI_Recv(source=4) MPI_Send(dest=4) Here, continue with examples using multiple communicators. Make sense? Let me know what you think, I'll post that (and more, to the FT wiki pages) Thanks, .Erez -----Original Message----- From: Narasimhan, Kannan [mailto:[email protected]] Sent: Tuesday, July 29, 2008 4:04 AM To: Erez Haba Cc: Narasimhan, Kannan Subject: MPI 3.0 FT WG: Error notification Hi Erez, Following up on our discussion in the MPI 3.0 FT Working group meeting --- Did U get to start on the error semantics/notification for a sample MPI collective call? Let me know if you want to bounce some options/ideas on this topic -- we can hash it out over email/phone call before this Friday's WG confcall. Thanx! Kannan _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Regular weekly call
by Richard Graham 03 Aug '08

03 Aug '08

In our last telecon we agreed to change the meeting frequency to once a week, to help move us forward at a faster rate. We will keep the meeting at the same time, rather than move it forward by an hour, as it accommodates a greater number of people. We will not meet the week of our face-to-face Forum meetings. Here are call details: Friday - 12-2 pm EDT, 5-7 pm GMT US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 Rich

1 0