January 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [mpi3-ft] Telecon on 2/1/2008
by Narasimhan, Kannan 25 Jan '08

25 Jan '08

Rich, One topic for the first meeting would be coming up with a clear definition/description of "Fault Tolerance" in the context of MPI. Obviously, we can refine it over email prior to teh confcall. Thanx! Kannan -----Original Message----- From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Richard Graham Sent: Friday, January 25, 2008 11:09 AM To: Discussion of MPI 3 Fault Tolerance Support Subject: [mpi3-ft] Telecon on 2/1/2008 Looking at the times people are available for this telecon, looks like Friday, 2/1/2008 from 12-2pm EST, 5-7pm GMT is the time that fits the most people who responded, so lets go with this time. We will also stick to this time slot on future calls, unless as a group we decide to change this. I know this does not work for all, so my apologies up front. What I really want to cover on this call are the use case scenarios we want to support, so we can have a target to shoot at. It you have some use cases you would like to go over, please do your best to send these out to the list by next wed, 1/30/2008, to give people time to look over these. Are there other items we need to discuss on this call ? Thanks, Rich _______________________________________________ mpi3-ft mailing list mpi3-ft(a)cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] Telecon on 2/1/2008
by Richard Graham 25 Jan '08

25 Jan '08

Forgot the call details: Toll free from within the US: 877-801-8130 Toll number: 1-203-692-8690 Code: 1044056 On 1/25/08 12:08 PM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: > Looking at the times people are available for this telecon, looks like > Friday, 2/1/2008 from 12-2pm EST, 5-7pm GMT is the time that fits the most > people who responded, so lets go with this time. We will also stick to this > time slot on future calls, unless as a group we decide to change this. I > know this does not work for all, so my apologies up front. > > What I really want to cover on this call are the use case scenarios we want > to support, so we can have a target to shoot at. It you have some use cases > you would like to go over, please do your best to send these out to the list > by next wed, 1/30/2008, to give people time to look over these. > > Are there other items we need to discuss on this call ? > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

[mpi3-ft] Telecon on 2/1/2008
by Richard Graham 25 Jan '08

25 Jan '08

Looking at the times people are available for this telecon, looks like Friday, 2/1/2008 from 12-2pm EST, 5-7pm GMT is the time that fits the most people who responded, so lets go with this time. We will also stick to this time slot on future calls, unless as a group we decide to change this. I know this does not work for all, so my apologies up front. What I really want to cover on this call are the use case scenarios we want to support, so we can have a target to shoot at. It you have some use cases you would like to go over, please do your best to send these out to the list by next wed, 1/30/2008, to give people time to look over these. Are there other items we need to discuss on this call ? Thanks, Rich

1 0

Re: [mpi3-ft] Picking up working group activities
by Josh Hursey 22 Jan '08

22 Jan '08

On Jan 22, 2008, at 11:22 AM, Richard Graham wrote: > > Wed: 10am ï¿½ 12pm EST/3pm ï¿½ 5pm GMT - Prefer Not > 11am ï¿½ 1pm EST/4pm ï¿½ 6pm GMT - Cannot > 12pm ï¿½ 2pm EST/5pm ï¿½ 8pm GMT - Cannot > Thur: 10am ï¿½ 12pm EST/3pm ï¿½ 5pm GMT - Good > 11am ï¿½ 1pm EST/4pm ï¿½ 6pm GMT - Good > 12pm ï¿½ 2pm EST/5pm ï¿½ 8pm GMT - Good > Fri: 10am ï¿½ 12pm EST/3pm ï¿½ 5pm GMT - Good > 11am ï¿½ 1pm EST/4pm ï¿½ 6pm GMT - Good > 12pm ï¿½ 2pm EST/5pm ï¿½ 8pm GMT - Good >

1 0

Re: [mpi3-ft] Picking up working group activities
by Narasimhan, Kannan 22 Jan '08

22 Jan '08

I've conflicts for the following slots: Wed: 12 pm - 2pm EST/3pm - 5pm GMT Thur: 10am - 12pm EST/3pm - 5pm GMT I'll be available for all other proposed time-slots (though my preference would be for the 11AM EST time-slots on Wed or Friday). Thanx! Kannan ________________________________ From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Richard Graham Sent: Tuesday, January 22, 2008 10:22 AM To: Discussion of MPI 3 Fault Tolerance Support Subject: Re: [mpi3-ft] Picking up working group activities Based on the feedback, I will propose the following times as potential meeting times: Wed: 10am - 12pm EST/3pm - 5pm GMT 11am - 1pm EST/4pm - 6pm GMT 12pm - 2pm EST/5pm - 8pm GMT Thur: 10am - 12pm EST/3pm - 5pm GMT 11am - 1pm EST/4pm - 6pm GMT 12pm - 2pm EST/5pm - 8pm GMT Fri: 10am - 12pm EST/3pm - 5pm GMT 11am - 1pm EST/4pm - 6pm GMT 12pm - 2pm EST/5pm - 8pm GMT I would expect this to be a standing meeting, occurring every 2 weeks, with the exception of the the weeks we have the face-to-face meetings. Please send your preferences, and we I will do the best to accommodate the greatest number of people we can. Rich On 1/21/08 1:14 PM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: I would like to start bi-weekly con calls to discuss Fault Tolerance and dynamic process support in the context of MPI 3.0. First, we need to find a time for the telecon that works for most people, so I will start by suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. How does this work for people who plan to be active participants in this work ? The items I would like to start addressing next week are - Does MPI provide FT, or enable FT work ? - Do we need more than is available in the current standard ? - If so, what are the use-case scenarios that we are aiming to support ? These seem to be a good starting point for discussions. If you have a particular use-case you want us to consider in this working group, please write it up and circulate it to on the mailing list at least a day before we meet, to give people a chance to read these over. Please keep the descriptions high-level, avoiding specific solutions, at this stage. We will start to discuss specifics once we have defined the scope of the problem that needs to be addressed - if we agree that there is one in the first place. Same thing for dynamic process support - if you have specific use-case scenarios that are not supported under the current standard, and you would like to have these considered for inclusion in the 3.0 standard, please provide the use cases you would like the group to consider. We should continue to take input on use case scenarios until the next face-to-face meeting, but the sooner we get these, the more effective we will be in proceeding in a timely manner. Thanks, Rich _______________________________________________ mpi3-ft mailing list mpi3-ft(a)cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] Picking up working group activities
by Erez Haba 22 Jan '08

22 Jan '08

I've listed my preferences. The 12pm slot would work for me any day. Wed: 10am - 12pm EST/3pm - 5pm GMT Cannot 11am - 1pm EST/4pm - 6pm GMT Prefer Not 12pm - 2pm EST/5pm - 8pm GMT Good Thur: 10am - 12pm EST/3pm - 5pm GMT Cannot 11am - 1pm EST/4pm - 6pm GMT Prefer Not 12pm - 2pm EST/5pm - 8pm GMT Good Fri: 10am - 12pm EST/3pm - 5pm GMT Cannot 11am - 1pm EST/4pm - 6pm GMT Prefer Not 12pm - 2pm EST/5pm - 8pm GMT Good From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Tuesday, January 22, 2008 10:03 AM To: Discussion of MPI 3 Fault Tolerance Support Subject: Re: [mpi3-ft] Picking up working group activities I added a column on the right with my preferences about the times. I'd like to avoid the 10am slots if possible since that is 7am here but if its necessary, I'll get up :) Wed: 10am - 12pm EST/3pm - 5pm GMT Prefer not 11am - 1pm EST/4pm - 6pm GMT Cannot 12pm - 2pm EST/5pm - 8pm GMT Good Thur: 10am - 12pm EST/3pm - 5pm GMT Good 11am - 1pm EST/4pm - 6pm GMT Good 12pm - 2pm EST/5pm - 8pm GMT Prefer Not Fri: 10am - 12pm EST/3pm - 5pm GMT Good 11am - 1pm EST/4pm - 6pm GMT Good 12pm - 2pm EST/5pm - 8pm GMT Good Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [mpi3-ft] Notes from 15 Jan. 2008 Meeting
by Greg Bronevetsky 22 Jan '08

22 Jan '08

>* Replication techniques should be added to the (presented) list of >possible recovery techniques to consider supporting. > - Question: Does considering replication techniques undermine the >HPC target community of the MPI standard. > - MPI interface should allow for this, maybe as an opt'in >functionality. Replication of MPI tasks? Is this really necessary? It seems to me that if you're going to go for something this expensive, it would be simple to add replication as a layer above MPI. >* An interface to easily piggyback data on point-to-point (and maybe >collective) messages would help support some high level checkpoint/ >restart techniques. We're working on putting together a piggybacking proposal. Checkpointing protocols are one of the use-cases. >* Should we expose to the application the fault recovery mechanism >used? (i.e., Network path failover) > - As a notification once complete? > - Or as a request to authorize the recovery mechanism before it >executes? We can only do that if we have a list of well-defined recovery mechanisms that we can present in the spec. I doubt that anybody is going to commit to such a list, so the only thing that we may be able to get away with are classes of recovery mechanisms: network recovery, node recovery, etc. I'm not sure what that user would be able to do with such information. They probably mostly want to know how long it will take and what MPI level objects will not be recovered (ex: the state of the dead process). >* Some users have expressed an interest in having an explicit >checkpoint function to identify 'good' times in the code execution to >checkpoint. Ideally when the checkpoint state is minimal. I don't think that the MPI forum should be in the business of defining a checkpointing API. Checkpointing is a whole-application operation, whereas MPI is just a library. Can you imagine the chaos that would ensue if every library decided on its own checkpointing API? The user would need to call each library's own relevant calls any time they wanted to do anything checkpoint-related. To keep things sane the user must be provided by the whole-application checkpointer with a single API that they will use for their checkpointing needs. There is no reason to make them issue MPI-specific checkpointing calls unless we can come up with some MPI-specific optimizations that users must be directly involved in. I'm skeptical about this possibility. We could of course define a couple of functions with the explicit intent that they should be used by the whole-application checkpointer, rather than the user. However, most prior work on MPI checkpointing has either modified MPI itself or has worked without any modifications to the MPI spec, meaning that there is little motivation to create such an API. >* Can we ask the MPI to save some internal state around a user level >checkpoint operation such that on recovery MPI objects such as >datatypes or communicators are automatically recreated for the user >before the application resumes execution. This can be efficiently done by a layer above MPI, so I'm not sure that such functionality needs to be in the spec. >* Collective communication: > - For some collectives a process can be in the collective or on >either side of it, how do the semantics of the collective operation >change in response to a failure/recovery operation. > - Bcast, for example, may allow some processes to exit early >providing some performance benefits. A fault tolerance semantic for >Bcast may require a global synchronization at the end of the Bcast to >ensure a two-phase commit of the effected buffers. > - We could consider loosening the synchronization constraints such >that the user could choose, via an MPI API, the level of confidence >they require for a process to exit a collective call. > - Maybe introduce the notion of epochs that can be started and >ended to help in fault isolation. Here's another option: if a collective fails, each receiver of the collective either receives the data or aborts (i.e. no hangs or invalid data). If the user is employing other fault tolerance techniques such as checkpointing or message logging, this functionality should be sufficient. >* Could fault tolerance only apply to a subset of the MPI API for >simplicity? For example, exclude collective communications or MPI I/O. Most applications use at least point-to-points or collectives, so if you disable fault tolerance for collectives, you disable it for most applications. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [mpi3-ft] Picking up working group activities
by Greg Bronevetsky 22 Jan '08

22 Jan '08

I added a column on the right with my preferences about the times. I'd like to avoid the 10am slots if possible since that is 7am here but if its necessary, I'll get up :) >Wed: 10am 12pm EST/3pm 5pm GMT Prefer not > 11am 1pm EST/4pm 6pm GMT Cannot > 12pm 2pm EST/5pm 8pm GMT Good >Thur: 10am 12pm EST/3pm 5pm GMT Good > 11am 1pm EST/4pm 6pm GMT Good > 12pm 2pm EST/5pm 8pm GMT Prefer Not >Fri: 10am 12pm EST/3pm 5pm GMT Good > 11am 1pm EST/4pm 6pm GMT Good > 12pm 2pm EST/5pm 8pm GMT Good Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [mpi3-ft] Picking up working group activities
by Richard Graham 22 Jan '08

22 Jan '08

Based on the feedback, I will propose the following times as potential meeting times: Wed: 10am 12pm EST/3pm 5pm GMT 11am 1pm EST/4pm 6pm GMT 12pm 2pm EST/5pm 8pm GMT Thur: 10am 12pm EST/3pm 5pm GMT 11am 1pm EST/4pm 6pm GMT 12pm 2pm EST/5pm 8pm GMT Fri: 10am 12pm EST/3pm 5pm GMT 11am 1pm EST/4pm 6pm GMT 12pm 2pm EST/5pm 8pm GMT I would expect this to be a standing meeting, occurring every 2 weeks, with the exception of the the weeks we have the face-to-face meetings. Please send your preferences, and we I will do the best to accommodate the greatest number of people we can. Rich On 1/21/08 1:14 PM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: > I would like to start bi-weekly con calls to discuss Fault Tolerance and > dynamic process support in the context of MPI 3.0. First, we need to find > a time for the telecon that works for most people, so I will start by > suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. > How does this work for people who plan to be active participants in this > work ? > > The items I would like to start addressing next week are > - Does MPI provide FT, or enable FT work ? > - Do we need more than is available in the current standard ? > - If so, what are the use-case scenarios that we are aiming to support ? > > These seem to be a good starting point for discussions. If you have a > particular use-case you want us to consider in this working group, please > write it up and circulate it to on the mailing list at least a day before we > meet, to give people a chance to read these over. Please keep the > descriptions high-level, avoiding specific solutions, at this stage. We > will start to discuss specifics once we have defined the scope of the > problem that needs to be addressed - if we agree that there is one in the > first place. > > Same thing for dynamic process support - if you have specific use-case > scenarios that are not supported under the current standard, and you would > like to have these considered for inclusion in the 3.0 standard, please > provide the use cases you would like the group to consider. > > We should continue to take input on use case scenarios until the next > face-to-face meeting, but the sooner we get these, the more effective we > will be in proceeding in a timely manner. > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft >

1 0

Re: [mpi3-ft] Picking up working group activities
by Narasimhan, Kannan 22 Jan '08

22 Jan '08

The proposed time-slot works for me (I'm in the CST timezone). As discussed (a little) at the MPI Forum meeting, I believe the standard must provide the infrastructure (i.e APIs) for implementations to "enable" FT, but stop short of mandating which specific FT features must be enabled. BTW, a unambiguous definition of FT in the context of MPI is probably the first thing that we need to come up with (this will also help us to scope our efforts). Thanx! Kannan -----Original Message----- From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Richard Graham Sent: Monday, January 21, 2008 12:15 PM To: mpi3-ft(a)cs.uiuc.edu Subject: [mpi3-ft] Picking up working group activities I would like to start bi-weekly con calls to discuss Fault Tolerance and dynamic process support in the context of MPI 3.0. First, we need to find a time for the telecon that works for most people, so I will start by suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. How does this work for people who plan to be active participants in this work ? The items I would like to start addressing next week are - Does MPI provide FT, or enable FT work ? - Do we need more than is available in the current standard ? - If so, what are the use-case scenarios that we are aiming to support ? These seem to be a good starting point for discussions. If you have a particular use-case you want us to consider in this working group, please write it up and circulate it to on the mailing list at least a day before we meet, to give people a chance to read these over. Please keep the descriptions high-level, avoiding specific solutions, at this stage. We will start to discuss specifics once we have defined the scope of the problem that needs to be addressed - if we agree that there is one in the first place. Same thing for dynamic process support - if you have specific use-case scenarios that are not supported under the current standard, and you would like to have these considered for inclusion in the 3.0 standard, please provide the use cases you would like the group to consider. We should continue to take input on use case scenarios until the next face-to-face meeting, but the sooner we get these, the more effective we will be in proceeding in a timely manner. Thanks, Rich _______________________________________________ mpi3-ft mailing list mpi3-ft(a)cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0