January 2008 - mpiwg-ft - mailman3.cels.anl.gov

[mpi3-ft] FW: Telecon on 2/1/2008
by Richard Graham 31 Jan '08

31 Jan '08

Just a reminder. Rich ------ Forwarded Message From: Richard Graham <rlgraham(a)ornl.gov> Reply-To: Discussion of MPI 3 Fault Tolerance Support <mpi3-ft(a)cs.uiuc.edu> Date: Fri, 25 Jan 2008 12:24:12 -0500 To: Discussion of MPI 3 Fault Tolerance Support <mpi3-ft(a)cs.uiuc.edu> Conversation: [mpi3-ft] Telecon on 2/1/2008 Subject: Re: [mpi3-ft] Telecon on 2/1/2008 Forgot the call details: Toll free from within the US: 877-801-8130 Toll number: 1-203-692-8690 Code: 1044056 On 1/25/08 12:08 PM, "Richard Graham" <rlgraham(a)ornl.gov> wrote: > Looking at the times people are available for this telecon, looks like > Friday, 2/1/2008 from 12-2pm EST, 5-7pm GMT is the time that fits the most > people who responded, so lets go with this time. We will also stick to this > time slot on future calls, unless as a group we decide to change this. I > know this does not work for all, so my apologies up front. > > What I really want to cover on this call are the use case scenarios we want > to support, so we can have a target to shoot at. It you have some use cases > you would like to go over, please do your best to send these out to the list > by next wed, 1/30/2008, to give people time to look over these. > > Are there other items we need to discuss on this call ? > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft ------ End of Forwarded Message

1 0

Re: [mpi3-ft] Notes from 15 Jan. 2008 Meeting
by Richard Graham 31 Jan '08

31 Jan '08

Josh, Can't get this through. Jeff will post this on the web as soon as we have things set up, which should be next week. Does not help now ... Rich On 1/30/08 11:22 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: > Rich, > > Can you send around a pdf of the slides you presented during the MPI > forum FT working group meeting? These slides might be helpful when > looking through these notes. > > Thanks, > Josh > > On Jan 22, 2008, at 1:32 PM, Greg Bronevetsky wrote: > >> >>> * Replication techniques should be added to the (presented) list of >>> possible recovery techniques to consider supporting. >>> - Question: Does considering replication techniques undermine the >>> HPC target community of the MPI standard. >>> - MPI interface should allow for this, maybe as an opt'in >>> functionality. >> Replication of MPI tasks? Is this really necessary? It seems to me >> that if you're going to go for something this expensive, it would be >> simple to add replication as a layer above MPI. >> >>> * An interface to easily piggyback data on point-to-point (and maybe >>> collective) messages would help support some high level checkpoint/ >>> restart techniques. >> >> We're working on putting together a piggybacking proposal. >> Checkpointing protocols are one of the use-cases. >> >>> * Should we expose to the application the fault recovery mechanism >>> used? (i.e., Network path failover) >>> - As a notification once complete? >>> - Or as a request to authorize the recovery mechanism before it >>> executes? >> We can only do that if we have a list of well-defined recovery >> mechanisms that we can present in the spec. I doubt that anybody is >> going to commit to such a list, so the only thing that we may be able >> to get away with are classes of recovery mechanisms: network >> recovery, node recovery, etc. I'm not sure what that user would be >> able to do with such information. They probably mostly want to know >> how long it will take and what MPI level objects will not be >> recovered (ex: the state of the dead process). >> >>> * Some users have expressed an interest in having an explicit >>> checkpoint function to identify 'good' times in the code execution to >>> checkpoint. Ideally when the checkpoint state is minimal. >> I don't think that the MPI forum should be in the business of >> defining a checkpointing API. Checkpointing is a whole-application >> operation, whereas MPI is just a library. Can you imagine the chaos >> that would ensue if every library decided on its own checkpointing >> API? The user would need to call each library's own relevant calls >> any time they wanted to do anything checkpoint-related. To keep >> things sane the user must be provided by the whole-application >> checkpointer with a single API that they will use for their >> checkpointing needs. There is no reason to make them issue >> MPI-specific checkpointing calls unless we can come up with some >> MPI-specific optimizations that users must be directly involved in. >> I'm skeptical about this possibility. >> >> We could of course define a couple of functions with the explicit >> intent that they should be used by the whole-application >> checkpointer, rather than the user. However, most prior work on MPI >> checkpointing has either modified MPI itself or has worked without >> any modifications to the MPI spec, meaning that there is little >> motivation to create such an API. >> >>> * Can we ask the MPI to save some internal state around a user level >>> checkpoint operation such that on recovery MPI objects such as >>> datatypes or communicators are automatically recreated for the user >>> before the application resumes execution. >> This can be efficiently done by a layer above MPI, so I'm not sure >> that such functionality needs to be in the spec. >> >>> * Collective communication: >>> - For some collectives a process can be in the collective or on >>> either side of it, how do the semantics of the collective operation >>> change in response to a failure/recovery operation. >>> - Bcast, for example, may allow some processes to exit early >>> providing some performance benefits. A fault tolerance semantic for >>> Bcast may require a global synchronization at the end of the Bcast to >>> ensure a two-phase commit of the effected buffers. >>> - We could consider loosening the synchronization constraints such >>> that the user could choose, via an MPI API, the level of confidence >>> they require for a process to exit a collective call. >>> - Maybe introduce the notion of epochs that can be started and >>> ended to help in fault isolation. >> Here's another option: if a collective fails, each receiver of the >> collective either receives the data or aborts (i.e. no hangs or >> invalid data). If the user is employing other fault tolerance >> techniques such as checkpointing or message logging, this >> functionality should be sufficient. >> >>> * Could fault tolerance only apply to a subset of the MPI API for >>> simplicity? For example, exclude collective communications or MPI I/ >>> O. >> Most applications use at least point-to-points or collectives, so if >> you disable fault tolerance for collectives, you disable it for most >> applications. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] Notes from 15 Jan. 2008 Meeting
by Richard Graham 31 Jan '08

31 Jan '08

resending On 1/30/08 11:22 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: > Rich, > > Can you send around a pdf of the slides you presented during the MPI > forum FT working group meeting? These slides might be helpful when > looking through these notes. > > Thanks, > Josh > > On Jan 22, 2008, at 1:32 PM, Greg Bronevetsky wrote: > >> >>> * Replication techniques should be added to the (presented) list of >>> possible recovery techniques to consider supporting. >>> - Question: Does considering replication techniques undermine the >>> HPC target community of the MPI standard. >>> - MPI interface should allow for this, maybe as an opt'in >>> functionality. >> Replication of MPI tasks? Is this really necessary? It seems to me >> that if you're going to go for something this expensive, it would be >> simple to add replication as a layer above MPI. >> >>> * An interface to easily piggyback data on point-to-point (and maybe >>> collective) messages would help support some high level checkpoint/ >>> restart techniques. >> >> We're working on putting together a piggybacking proposal. >> Checkpointing protocols are one of the use-cases. >> >>> * Should we expose to the application the fault recovery mechanism >>> used? (i.e., Network path failover) >>> - As a notification once complete? >>> - Or as a request to authorize the recovery mechanism before it >>> executes? >> We can only do that if we have a list of well-defined recovery >> mechanisms that we can present in the spec. I doubt that anybody is >> going to commit to such a list, so the only thing that we may be able >> to get away with are classes of recovery mechanisms: network >> recovery, node recovery, etc. I'm not sure what that user would be >> able to do with such information. They probably mostly want to know >> how long it will take and what MPI level objects will not be >> recovered (ex: the state of the dead process). >> >>> * Some users have expressed an interest in having an explicit >>> checkpoint function to identify 'good' times in the code execution to >>> checkpoint. Ideally when the checkpoint state is minimal. >> I don't think that the MPI forum should be in the business of >> defining a checkpointing API. Checkpointing is a whole-application >> operation, whereas MPI is just a library. Can you imagine the chaos >> that would ensue if every library decided on its own checkpointing >> API? The user would need to call each library's own relevant calls >> any time they wanted to do anything checkpoint-related. To keep >> things sane the user must be provided by the whole-application >> checkpointer with a single API that they will use for their >> checkpointing needs. There is no reason to make them issue >> MPI-specific checkpointing calls unless we can come up with some >> MPI-specific optimizations that users must be directly involved in. >> I'm skeptical about this possibility. >> >> We could of course define a couple of functions with the explicit >> intent that they should be used by the whole-application >> checkpointer, rather than the user. However, most prior work on MPI >> checkpointing has either modified MPI itself or has worked without >> any modifications to the MPI spec, meaning that there is little >> motivation to create such an API. >> >>> * Can we ask the MPI to save some internal state around a user level >>> checkpoint operation such that on recovery MPI objects such as >>> datatypes or communicators are automatically recreated for the user >>> before the application resumes execution. >> This can be efficiently done by a layer above MPI, so I'm not sure >> that such functionality needs to be in the spec. >> >>> * Collective communication: >>> - For some collectives a process can be in the collective or on >>> either side of it, how do the semantics of the collective operation >>> change in response to a failure/recovery operation. >>> - Bcast, for example, may allow some processes to exit early >>> providing some performance benefits. A fault tolerance semantic for >>> Bcast may require a global synchronization at the end of the Bcast to >>> ensure a two-phase commit of the effected buffers. >>> - We could consider loosening the synchronization constraints such >>> that the user could choose, via an MPI API, the level of confidence >>> they require for a process to exit a collective call. >>> - Maybe introduce the notion of epochs that can be started and >>> ended to help in fault isolation. >> Here's another option: if a collective fails, each receiver of the >> collective either receives the data or aborts (i.e. no hangs or >> invalid data). If the user is employing other fault tolerance >> techniques such as checkpointing or message logging, this >> functionality should be sufficient. >> >>> * Could fault tolerance only apply to a subset of the MPI API for >>> simplicity? For example, exclude collective communications or MPI I/ >>> O. >> Most applications use at least point-to-points or collectives, so if >> you disable fault tolerance for collectives, you disable it for most >> applications. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] Notes from 15 Jan. 2008 Meeting
by Josh Hursey 30 Jan '08

30 Jan '08

Rich, Can you send around a pdf of the slides you presented during the MPI forum FT working group meeting? These slides might be helpful when looking through these notes. Thanks, Josh On Jan 22, 2008, at 1:32 PM, Greg Bronevetsky wrote: > >> * Replication techniques should be added to the (presented) list of >> possible recovery techniques to consider supporting. >> - Question: Does considering replication techniques undermine the >> HPC target community of the MPI standard. >> - MPI interface should allow for this, maybe as an opt'in >> functionality. > Replication of MPI tasks? Is this really necessary? It seems to me > that if you're going to go for something this expensive, it would be > simple to add replication as a layer above MPI. > >> * An interface to easily piggyback data on point-to-point (and maybe >> collective) messages would help support some high level checkpoint/ >> restart techniques. > > We're working on putting together a piggybacking proposal. > Checkpointing protocols are one of the use-cases. > >> * Should we expose to the application the fault recovery mechanism >> used? (i.e., Network path failover) >> - As a notification once complete? >> - Or as a request to authorize the recovery mechanism before it >> executes? > We can only do that if we have a list of well-defined recovery > mechanisms that we can present in the spec. I doubt that anybody is > going to commit to such a list, so the only thing that we may be able > to get away with are classes of recovery mechanisms: network > recovery, node recovery, etc. I'm not sure what that user would be > able to do with such information. They probably mostly want to know > how long it will take and what MPI level objects will not be > recovered (ex: the state of the dead process). > >> * Some users have expressed an interest in having an explicit >> checkpoint function to identify 'good' times in the code execution to >> checkpoint. Ideally when the checkpoint state is minimal. > I don't think that the MPI forum should be in the business of > defining a checkpointing API. Checkpointing is a whole-application > operation, whereas MPI is just a library. Can you imagine the chaos > that would ensue if every library decided on its own checkpointing > API? The user would need to call each library's own relevant calls > any time they wanted to do anything checkpoint-related. To keep > things sane the user must be provided by the whole-application > checkpointer with a single API that they will use for their > checkpointing needs. There is no reason to make them issue > MPI-specific checkpointing calls unless we can come up with some > MPI-specific optimizations that users must be directly involved in. > I'm skeptical about this possibility. > > We could of course define a couple of functions with the explicit > intent that they should be used by the whole-application > checkpointer, rather than the user. However, most prior work on MPI > checkpointing has either modified MPI itself or has worked without > any modifications to the MPI spec, meaning that there is little > motivation to create such an API. > >> * Can we ask the MPI to save some internal state around a user level >> checkpoint operation such that on recovery MPI objects such as >> datatypes or communicators are automatically recreated for the user >> before the application resumes execution. > This can be efficiently done by a layer above MPI, so I'm not sure > that such functionality needs to be in the spec. > >> * Collective communication: >> - For some collectives a process can be in the collective or on >> either side of it, how do the semantics of the collective operation >> change in response to a failure/recovery operation. >> - Bcast, for example, may allow some processes to exit early >> providing some performance benefits. A fault tolerance semantic for >> Bcast may require a global synchronization at the end of the Bcast to >> ensure a two-phase commit of the effected buffers. >> - We could consider loosening the synchronization constraints such >> that the user could choose, via an MPI API, the level of confidence >> they require for a process to exit a collective call. >> - Maybe introduce the notion of epochs that can be started and >> ended to help in fault isolation. > Here's another option: if a collective fails, each receiver of the > collective either receives the data or aborts (i.e. no hangs or > invalid data). If the user is employing other fault tolerance > techniques such as checkpointing or message logging, this > functionality should be sufficient. > >> * Could fault tolerance only apply to a subset of the MPI API for >> simplicity? For example, exclude collective communications or MPI I/ >> O. > Most applications use at least point-to-points or collectives, so if > you disable fault tolerance for collectives, you disable it for most > applications. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] FW: Starting slides for 2/1/2008 telecon
by Richard Graham 30 Jan '08

30 Jan '08

On 1/30/08 4:40 AM, "Thomas Herault" <thomas.herault(a)lri.fr> wrote: > > > Le 30 janv. 08 à 09:17, Greg Bronevetsky a écrit : > >> > Two comments: >> > >> > We may want to add the capability to spawn new processes and give >> > them the ranks of the failed processes. This is more efficient than >> > pre-allocating enough spare processes as part of the original job >> > allocation, so it might be a good idea to include in the spec. >> > > > A user should be able to spawn new processes and create a new > communicator with enough processes to circumvent this issue. > > I think that size of MPI_COMM_WORLD should decrease > when failures occur. Not all fault tolerance solutions need processor > replacement (e.g. a master/worker approach when a worker fails). > But MPI_UNIVERSE_SIZE should be allowed to remain constant > (thus, the system can provide dynamically new processors to > replace failed ones). > >>> >> No disagreements. As I mentioned earlier, we need to start with the >>> >> non-trivial amount of work that has already been done in this area. At >>> >> this stage I am deliberately staying away from trying to propose specific >>> >> solutions we need to first define what we think we can address, and >>> >> what we will just pass on. > >> > I disagree with the comments about MPI quieting the communication >> > system because this presumes that the application will use the >> > trivial sync-and-stop CPR protocol. They may the case but we >> > shouldn't write this assumption into the spec. We should probably >> > restrict ourselves to only saying that no message may get partially >> > delivered since such messages would be very hard to deal with above >> > the MPI library. >> > > > I agree that we should not assume that synchronized CPR will be the > only approach used. Messages must certainly be kept transactional > (either completely received or not at all). But what about collective > communications? As suggested during the meeting, a two-phase commit > protocol can be enforced to ensure that any collective communication > either completes or fails on any living processor; however this may be > considered as too inefficient for the normal case, when failures do not > occur. > >>> >> No assumption on solution is being made. At this stage I believe that >>> the >>> >> primitives listed will cover the current solutions people have deployed. >>> >> The intent is to provide as set of tools that can be used as needed >>> i.e. >>> >> the standard should not be advocating a solution, but enabling solutions >>> >> that don¹t involve modifying every MPI implementation (if possible). If >>> there >>> >> are missing primitives, please bring them up. If there are primitives >>> listed >>> >> that are not needed, please mention these too. > > Rich > > Thomas Herault > assistant professor > INRIA/Univ. Paris Sud > >> > Greg Bronevetsky >> > Post-Doctoral Researcher >> > 1028 Building 451 >> > Lawrence Livermore National Lab >> > (925) 424-5756 >> > bronevetsky1(a)llnl.gov >> > >> > At 10:12 AM 1/29/2008, Richard Graham wrote: >>> >> This did not seem to make it through the first time, so let me try >>> >> again. >>> >> >>> >> Rich >>> >> >>> >> ------ Forwarded Message >>> >> From: Richard Graham <rlgraham(a)ornl.gov> >>> >> Date: Tue, 29 Jan 2008 10:55:11 -0500 >>> >> To: Discussion of MPI 3 Fault Tolerance Support <mpi3-ft(a)cs.uiuc.edu> >>> >> Conversation: Starting slides for 2/1/2008 telecon >>> >> Subject: Starting slides for 2/1/2008 telecon >>> >> >>> >> Attached is a set of slides I intend to use as a staring point for >>> >> the >>> >> telecon this coming Friday. If you are planning on attending, >>> >> please take a >>> >> look at these, and see what is missing. The main goal for this >>> >> call is to >>> >> help set the scope of the problem for which we intend to propose >>> >> changes to >>> >> the MPI standard. >>> >> >>> >> Thanks, >>> >> Rich >>> >> >>> >> ------ End of Forwarded Message >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> mpi3-ft mailing list >>> >> mpi3-ft(a)cs.uiuc.edu >>> >> http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft >> > _______________________________________________ >> > mpi3-ft mailing list >> > mpi3-ft(a)cs.uiuc.edu >> > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft >

1 0

Re: [mpi3-ft] FW: Starting slides for 2/1/2008 telecon
by Richard Graham 30 Jan '08

30 Jan '08

On 1/30/08 3:17 AM, "Greg Bronevetsky" <bronevetsky1(a)llnl.gov> wrote: > Two comments: > > We may want to add the capability to spawn new processes and give > them the ranks of the failed processes. This is more efficient than > pre-allocating enough spare processes as part of the original job > allocation, so it might be a good idea to include in the spec. > >>> >> At this stage no solutions are being proposed. We need to figure out >>> what >>> >> sort of scenarios we aim to provide assistance for, before we start >>> proposing >>> >> solutions, or even considering API¹s > > I disagree with the comments about MPI quieting the communication > system because this presumes that the application will use the > trivial sync-and-stop CPR protocol. They may the case but we > shouldn't write this assumption into the spec. We should probably > restrict ourselves to only saying that no message may get partially > delivered since such messages would be very hard to deal with above > the MPI library. > >>> >> What is listed are things that people do NOW. This will obviously >>> include >>> >> things that you agree with, and things you disagree with, and is only the >>> >> starting point for discussion. The fact that I listed these, does not >>> mean >>> >> that I agree with them, it just means that this is part of the state of >>> the >>> >> art. > > Rich > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > At 10:12 AM 1/29/2008, Richard Graham wrote: >> >This did not seem to make it through the first time, so let me try again. >> > >> >Rich >> > >> >------ Forwarded Message >> >From: Richard Graham <rlgraham(a)ornl.gov> >> >Date: Tue, 29 Jan 2008 10:55:11 -0500 >> >To: Discussion of MPI 3 Fault Tolerance Support <mpi3-ft(a)cs.uiuc.edu> >> >Conversation: Starting slides for 2/1/2008 telecon >> >Subject: Starting slides for 2/1/2008 telecon >> > >> >Attached is a set of slides I intend to use as a staring point for the >> >telecon this coming Friday. If you are planning on attending, please take a >> >look at these, and see what is missing. The main goal for this call is to >> >help set the scope of the problem for which we intend to propose changes to >> >the MPI standard. >> > >> >Thanks, >> >Rich >> > >> >------ End of Forwarded Message >> > >> > >> > >> >_______________________________________________ >> >mpi3-ft mailing list >> >mpi3-ft(a)cs.uiuc.edu >> >http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft >

1 0

Re: [mpi3-ft] FW: Starting slides for 2/1/2008 telecon
by Thomas Herault 30 Jan '08

30 Jan '08

Le 30 janv. 08 ï¿½ 09:17, Greg Bronevetsky a ï¿½crit : > Two comments: > > We may want to add the capability to spawn new processes and give > them the ranks of the failed processes. This is more efficient than > pre-allocating enough spare processes as part of the original job > allocation, so it might be a good idea to include in the spec. > A user should be able to spawn new processes and create a new communicator with enough processes to circumvent this issue. I think that size of MPI_COMM_WORLD should decrease when failures occur. Not all fault tolerance solutions need processor replacement (e.g. a master/worker approach when a worker fails). But MPI_UNIVERSE_SIZE should be allowed to remain constant (thus, the system can provide dynamically new processors to replace failed ones). > I disagree with the comments about MPI quieting the communication > system because this presumes that the application will use the > trivial sync-and-stop CPR protocol. They may the case but we > shouldn't write this assumption into the spec. We should probably > restrict ourselves to only saying that no message may get partially > delivered since such messages would be very hard to deal with above > the MPI library. > I agree that we should not assume that synchronized CPR will be the only approach used. Messages must certainly be kept transactional (either completely received or not at all). But what about collective communications? As suggested during the meeting, a two-phase commit protocol can be enforced to ensure that any collective communication either completes or fails on any living processor; however this may be considered as too inefficient for the normal case, when failures do not occur. Thomas Herault assistant professor INRIA/Univ. Paris Sud > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > At 10:12 AM 1/29/2008, Richard Graham wrote: >> This did not seem to make it through the first time, so let me try >> again. >> >> Rich >> >> ------ Forwarded Message >> From: Richard Graham <rlgraham(a)ornl.gov> >> Date: Tue, 29 Jan 2008 10:55:11 -0500 >> To: Discussion of MPI 3 Fault Tolerance Support <mpi3-ft(a)cs.uiuc.edu> >> Conversation: Starting slides for 2/1/2008 telecon >> Subject: Starting slides for 2/1/2008 telecon >> >> Attached is a set of slides I intend to use as a staring point for >> the >> telecon this coming Friday. If you are planning on attending, >> please take a >> look at these, and see what is missing. The main goal for this >> call is to >> help set the scope of the problem for which we intend to propose >> changes to >> the MPI standard. >> >> Thanks, >> Rich >> >> ------ End of Forwarded Message >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] FW: Starting slides for 2/1/2008 telecon
by Greg Bronevetsky 30 Jan '08

30 Jan '08

Two comments: We may want to add the capability to spawn new processes and give them the ranks of the failed processes. This is more efficient than pre-allocating enough spare processes as part of the original job allocation, so it might be a good idea to include in the spec. I disagree with the comments about MPI quieting the communication system because this presumes that the application will use the trivial sync-and-stop CPR protocol. They may the case but we shouldn't write this assumption into the spec. We should probably restrict ourselves to only saying that no message may get partially delivered since such messages would be very hard to deal with above the MPI library. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 10:12 AM 1/29/2008, Richard Graham wrote: >This did not seem to make it through the first time, so let me try again. > >Rich > >------ Forwarded Message >From: Richard Graham <rlgraham(a)ornl.gov> >Date: Tue, 29 Jan 2008 10:55:11 -0500 >To: Discussion of MPI 3 Fault Tolerance Support <mpi3-ft(a)cs.uiuc.edu> >Conversation: Starting slides for 2/1/2008 telecon >Subject: Starting slides for 2/1/2008 telecon > >Attached is a set of slides I intend to use as a staring point for the >telecon this coming Friday. If you are planning on attending, please take a >look at these, and see what is missing. The main goal for this call is to >help set the scope of the problem for which we intend to propose changes to >the MPI standard. > >Thanks, >Rich > >------ End of Forwarded Message > > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)cs.uiuc.edu >http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

[mpi3-ft] FW: Starting slides for 2/1/2008 telecon
by Richard Graham 29 Jan '08

29 Jan '08

This did not seem to make it through the first time, so let me try again. Rich ------ Forwarded Message From: Richard Graham <rlgraham(a)ornl.gov> Date: Tue, 29 Jan 2008 10:55:11 -0500 To: Discussion of MPI 3 Fault Tolerance Support <mpi3-ft(a)cs.uiuc.edu> Conversation: Starting slides for 2/1/2008 telecon Subject: Starting slides for 2/1/2008 telecon Attached is a set of slides I intend to use as a staring point for the telecon this coming Friday. If you are planning on attending, please take a look at these, and see what is missing. The main goal for this call is to help set the scope of the problem for which we intend to propose changes to the MPI standard. Thanks, Rich ------ End of Forwarded Message

1 0

Re: [mpi3-ft] Telecon on 2/1/2008
by Greg Bronevetsky 25 Jan '08

25 Jan '08

>One topic for the first meeting would be coming up with a clear >definition/description of "Fault Tolerance" in the context of MPI. >Obviously, we can refine it over email prior to teh confcall. More specifically, I think we need to break the overall task of application fault tolerance into application responsibilities, MPI responsibilities and third party responsibilities. For example, an MPI responsibility may be to maintain stability in the face of non-catastrophic node and link failures (i.e. most nodes are alive and no network partition). An application responsibility may be to survive the failure of one or more nodes through some application-specific technique or through checkpointing. A third-party responsibility may be checkpointing, which may be included as part of the application or embedded into MPI but will probably not be specified as part of the MPI spec. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0