January 2009 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Conference Call Today?
by Richard Graham 28 Jan '09

28 Jan '09

There is - I forgot sorry, just getting on On 1/28/09 12:12 PM, "Erez Haba" <erezh(a)MICROSOFT.com> wrote: > I'm waiting on the call. Listening to the music :) > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Josh Hursey > Sent: Wednesday, January 28, 2009 9:11 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] Conference Call Today? > > Do we have a teleconf today? > > -- Josh > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Quiescence Interface
by Josh Hursey 28 Jan '09

28 Jan '09

Starting discussion on this proposal at the next forum meeting sounds fine to me. I don't know if we need to devote the whole time to this proposal, but I can prepare a few things for our discussion. I hope to have a basic prototype ready by then too. I can contact Paul Hargrove (BLCR) to see if he would be interested in attending this discussion. Since this proposal is a collective operation, it might be useful to have someone from the collective working group sit in to make sure the proposal is phrased appropriately. Cheers, Josh On Jan 28, 2009, at 10:22 AM, Richard Graham wrote: > Josh, > Thanks for sending this around. > I would like to schedule a time to start discussion this > proposal. To do > so, I think we need to have the impacted parties at the table. > Since the > Evergrid (sp) folks and the BLCR folks are in the Bay area, would > it make > sense to devote our working group time at the Forum to this > subject, and > invite then to attend ? Who else would we want to have participate > in this > discussion ? > > Thanks, > Rich > > > On 1/22/09 10:21 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: > >> I have updated the Quiescence Interface proposal on the wiki: >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Quiescence >> >> I am interested in questions and comments from the group regarding >> the >> interface, and use-cases. The primary use-case is for application >> initiated checkpoint/restart, but the semantics of the proposed API >> likely also open doors to other use-cases. >> >> Let me know what you think. >> >> Cheers, >> Josh >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Conference Call Today?
by Erez Haba 28 Jan '09

28 Jan '09

I'm waiting on the call. Listening to the music :) -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Wednesday, January 28, 2009 9:11 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Conference Call Today? Do we have a teleconf today? -- Josh _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Concall today?
by Narasimhan, Kannan 28 Jan '09

28 Jan '09

Waiting on a music loop -- are we on for today's FT Working Group conf call? Thanx! Kannan

1 0

[Mpi3-ft] Conference Call Today?
by Josh Hursey 28 Jan '09

28 Jan '09

Do we have a teleconf today? -- Josh

1 0

[Mpi3-ft] Are we having the telecon today?
by Greg Bronevetsky 28 Jan '09

28 Jan '09

Are we having the telecon today? Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Quiescence Interface
by Joseph Ruscio 28 Jan '09

28 Jan '09

I could most likely attend on Evergrid (now Librato)'s behalf, and Mike and I will put together some review comments from a checkpoint implementor's point of view. thanks, Joe On Jan 28, 2009, at 7:22 AM, Richard Graham wrote: > Josh, > Thanks for sending this around. > I would like to schedule a time to start discussion this proposal. > To do > so, I think we need to have the impacted parties at the table. > Since the > Evergrid (sp) folks and the BLCR folks are in the Bay area, would it > make > sense to devote our working group time at the Forum to this subject, > and > invite then to attend ? Who else would we want to have participate > in this > discussion ? > > Thanks, > Rich > > > On 1/22/09 10:21 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: > >> I have updated the Quiescence Interface proposal on the wiki: >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Quiescence >> >> I am interested in questions and comments from the group regarding >> the >> interface, and use-cases. The primary use-case is for application >> initiated checkpoint/restart, but the semantics of the proposed API >> likely also open doors to other use-cases. >> >> Let me know what you think. >> >> Cheers, >> Josh >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Quiescence Interface
by Richard Graham 28 Jan '09

28 Jan '09

Josh, Thanks for sending this around. I would like to schedule a time to start discussion this proposal. To do so, I think we need to have the impacted parties at the table. Since the Evergrid (sp) folks and the BLCR folks are in the Bay area, would it make sense to devote our working group time at the Forum to this subject, and invite then to attend ? Who else would we want to have participate in this discussion ? Thanks, Rich On 1/22/09 10:21 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: > I have updated the Quiescence Interface proposal on the wiki: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Quiescence > > I am interested in questions and comments from the group regarding the > interface, and use-cases. The primary use-case is for application > initiated checkpoint/restart, but the semantics of the proposed API > likely also open doors to other use-cases. > > Let me know what you think. > > Cheers, > Josh > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Quiescence Interface
by Greg Bronevetsky 28 Jan '09

28 Jan '09

First, Josh, your proposal has overcome my biggest objection: it doesn't specify things that cannot be specified in MPI and is therefore well-defined. However, having spent some time thinking about it, there is something that doesn't make sense to me. The quiescence API states that all MPI communication must be above a certain level of abstraction. Lets say for the sake of argument that this is the level between MPI and Infiniband Verbs or sockets, meaning that all messages are in RAM and not on the network card. However, even in this case this is not sufficient for recovery because on restart the IDs of the nodes the application is running on will be different from those in the original execution. Thus, MPI will restart normally and any communication that uses the old node IDs will fail. The same applies to other entities and resources such as pinned memory, Infiniband connections, etc. As such, we really have two options. - Require MPI to essentially create a restart image of itself so that on restart it can recreate all the data that changes, such as node IDs, pinned memory, connections, etc. This is quite demanding from the perspective of MPI developers but may be doable. (MPI implementors, please chime in here!) - Let the checkpointing library completely virtualize everything at the preferred abstraction level so that on restart even if MPI uses node IDs from its original execution, this virtualization layer will remap them to their new IDs on the fly. The same can be done for most other resources and entities that exist below the level of abstraction. However, if the checkpointing library can do that, there is not need for the quiescence API because the checkpointing library can ensure quiescence on its own by using the same mechanisms that MPI would have used. As such, it appears to me that we either force system-level checkpointing layers to virtualize everything on their own (already done by VMM-based checkpointers) or we provide a full checkpoint/restart API for MPI rather than a simple quiescence API. So the options are to do nothing or to demand everything, neither of which is appealing. Does this make sense? What do you guys think? Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 10:26 AM 1/22/2009, Josh Hursey wrote: >On Jan 22, 2009, at 11:02 AM, Greg Bronevetsky wrote: > >>At 07:21 AM 1/22/2009, Josh Hursey wrote: >>>I have updated the Quiescence Interface proposal on the wiki: >>> https:// svn.mpi-forum.org/trac/mpi-forum-web/wiki/Quiescence >>> >>>I am interested in questions and comments from the group regarding >>>the >>>interface, and use-cases. The primary use-case is for application >>>initiated checkpoint/restart, but the semantics of the proposed API >>>likely also open doors to other use-cases. >>> >>>Let me know what you think. >>I like that this proposal is more detailed than our previous >>discussions but I don't think that my major concern has been >>addressed yet. The key text in the proposal is "all of the in-flight >>messages have been accounted for in the specified comm". What does >>"accounted for" mean? Does the relevant message information go into >>some sort of buffer? Is it on the network card? What semantics are >>we actually guaranteeing here? > >I missed some of this text when copying it over to the wiki. I updated >the wiki with some of the wording below. > >'accounted for' means that any message sent by one process on the >specified communicator has been transmitted to the recipient. The >recipient may either buffer the message in the MPI implementation (if >no receive has been posted) or place it in the recipient's buffer (if >a receive has been posted). > >The MPI implementation can choose to buffer the message contents >however it determines best (network card, internal data structure, >specialized hardware). The MPI_Info argument set may allow an >application to express any requirements regarding how the MPI >implementation buffers the message contents with regard to MPI managed >devices such as the network card. I think specifying this type of a >key is implementation specific and should be left as an optional, MPI >implementation defined key. > >>I'm comfortable with the idea that the semantics are application >>dependent and this is just a standard API for applications to use to >>access these semantics but in that case the value of having this API >>is significantly reduced. > >I can see your point. For the system-level checkpointing use-case, an >MPI library that provides guarantees about the state of the network >cards during the quiescent region is important. > >I have heard from a couple of application developers that providing a >mechanism for 'fencing' all of the messages on a communicator in a >single call is also useful for purely application level checkpointing. >They just want a (better?) mechanism for forcing all messages to the >recipients in a synchronized manner. > > >So I believe that this API is useful even if the implementation >specific details of how the message contents are buffered by the >recipient are more loosely defined. > >I'm interested in hearing if others have suggestions on alternative >use cases where this API might be useful. > >One use-case might be an application that would like to prevent the >MPI library's progress engine from using CPU cycles for a region of >time. They would use this API to define a region of time in which it >may compute without interference from the MPI library's progress >engine looking for unexpected receives. Just a thought, but I don't >have a clear target application for this use-case so I did not include >it in the proposal. > >-- Josh > > > > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Quiescence Interface
by Josh Hursey 22 Jan '09

22 Jan '09

On Jan 22, 2009, at 11:02 AM, Greg Bronevetsky wrote: > At 07:21 AM 1/22/2009, Josh Hursey wrote: >> I have updated the Quiescence Interface proposal on the wiki: >> https:// svn.mpi-forum.org/trac/mpi-forum-web/wiki/Quiescence >> >> I am interested in questions and comments from the group regarding >> the >> interface, and use-cases. The primary use-case is for application >> initiated checkpoint/restart, but the semantics of the proposed API >> likely also open doors to other use-cases. >> >> Let me know what you think. > I like that this proposal is more detailed than our previous > discussions but I don't think that my major concern has been > addressed yet. The key text in the proposal is "all of the in-flight > messages have been accounted for in the specified comm". What does > "accounted for" mean? Does the relevant message information go into > some sort of buffer? Is it on the network card? What semantics are > we actually guaranteeing here? I missed some of this text when copying it over to the wiki. I updated the wiki with some of the wording below. 'accounted for' means that any message sent by one process on the specified communicator has been transmitted to the recipient. The recipient may either buffer the message in the MPI implementation (if no receive has been posted) or place it in the recipient's buffer (if a receive has been posted). The MPI implementation can choose to buffer the message contents however it determines best (network card, internal data structure, specialized hardware). The MPI_Info argument set may allow an application to express any requirements regarding how the MPI implementation buffers the message contents with regard to MPI managed devices such as the network card. I think specifying this type of a key is implementation specific and should be left as an optional, MPI implementation defined key. > I'm comfortable with the idea that the semantics are application > dependent and this is just a standard API for applications to use to > access these semantics but in that case the value of having this API > is significantly reduced. I can see your point. For the system-level checkpointing use-case, an MPI library that provides guarantees about the state of the network cards during the quiescent region is important. I have heard from a couple of application developers that providing a mechanism for 'fencing' all of the messages on a communicator in a single call is also useful for purely application level checkpointing. They just want a (better?) mechanism for forcing all messages to the recipients in a synchronized manner. So I believe that this API is useful even if the implementation specific details of how the message contents are buffered by the recipient are more loosely defined. I'm interested in hearing if others have suggestions on alternative use cases where this API might be useful. One use-case might be an application that would like to prevent the MPI library's progress engine from using CPU cycles for a region of time. They would use this API to define a region of time in which it may compute without interference from the MPI library's progress engine looking for unexpected receives. Just a thought, but I don't have a clear target application for this use-case so I did not include it in the proposal. -- Josh

1 0