October 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] system-level C/R requirements
by Thomas Herault 27 Oct '08

27 Oct '08

Le 27 oct. 08 ï¿½ 17:16, Joseph Ruscio a ï¿½crit : > > On Oct 27, 2008, at 7:18 AM, Thomas Herault wrote: > >> So, my question is: what are the benefits of solution b) as >> compared to solution a)? > > > If you go with solution b), the only thing the MPI implementation > needs to worry about is PREPAREing and RESTOREing the network state. > All of the checkpoint specific issues and complexity such as queuing > system integration, checkpoint file storage options like redundant > local storage, etc. > Except that, as pointed out before, what PREPARE and RESTORE must do is very system and checkpoint mechanism dependent. > In case a), MPI implementations need to concern themselves with > these issues AND support the non-standard checkpoint invocation > API's for every different desired system checkpointer. Agreed. The point of case a) is that we modify the MPI library knowing exactly what the checkpoint mechanism saves. Since we are handling the network specific parts, we can find workarounds for things like pinned pages, without too much impact on the network state (like having to completely close the network interface, to re-open it in RESTORE). > So that's a question of whether the responsibility for system-level > checkpointer integration lays on the MPI implementor or the > individual checkpoint implementors. > > Going with b) gives MPI an opportunity to minimally specify an > integration mechanism with system-level checkpointer's. If the MPI > implementor does not wish to worry about these classes of CP/R > systems, they just don't implement the minimal set of calls. If they > do want the support, they implement the calls. > I agree that the point of case b) would be to relieve the data movement/saving/restoring part. However, one could argue that MPI is also well-placed to do this part (using MPI-IO for example). I am not convinced that this will give an opportunity to *minimally* specify the integration mechanism. This would be minimal in the number of calls to implement, but the cost of these calls would be nothing but optimal/minimal. > A single implementation of PREPARE and RESTORE would support most > system-level checkpointers. For example our checkpointer that sits > completely in user-land and BLCR that sits completely in the OS > would have the same set of requirements. VMM level checkpointers > have been suggested as being different. Wouldn't they either have > the same requirements, or be completely transparent to the MPI stack > i.e. snapshotting the OS, communication device, etc and allowing the > protocol to sort out lost messages? > Moreover, to have an integration mechanism which support most system- level checkpointers, it seems to me that you imply that PREPARE/ RESTORE would be generic. Then, we need to define more precisely what they would do. The current proposition of Alexander is to have a vague definition (like PREPARE put the MPI in a checkpointable state, RESTORE restores communications, and MPI is authorized to do something that would not be checkpointable), and to let implementations find a way to implement this specification. We would end with implementation / checkpointing systems pairs, like Alexander said. So, I don't really see the benefit as compared to case a) (where we wold also have implementation / checkpointing systems pairs). > cheers, > Joe Cheers, Thomas

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Greg Bronevetsky 27 Oct '08

27 Oct '08

Before saying anything else I want to thank Josh for bringing this up. Checkpointing is one of the more complex use-cases of the MPI fault tolerance spec. As such, we should keep it in mind when we design the spec but not spend all our time on it. The list that Josh provided is overall more important than the fine issues of the specific checkpointing use-case. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 05:44 AM 10/27/2008, Josh Hursey wrote: >Just to throw my two cents in here. > >I think the question of how to support checkpoint/restart in MPI is >important. One question that I proposed for further thought is: > Why can't an application call into a system level C/R system >directly when running with MPI? Why does it need MPI's support here? >As a related, but maybe slightly less complex question: > What is hindering application level checkpoint/restart mechanisms >when running with MPI? > >The answer to the former question is a combination of possibly saving >internal state of the MPI, interconnect support, and message tracking >(and more). This is a complex issue, per the discussion on and off the >list so far. I like the spirit of some of the proposals, particularly >a simple, non-binding API to signal the intentions of the application. >I think these need more refinement, and I don't have a clear idea for >a solution yet. > >The answer to the latter is, in my opinion, not much. Applications >already take their own internal checkpoints and can restart from them >during a relaunch of MPI. If we rephrase the question to be: > How can we better support application level checkpoint/restart with >MPI? > >Then we have something we can explore. At large scale, relaunching the >entire application just to recover from the failure of a small subset >of process failures using an application level checkpoint may be too >time consuming (and largely overkill). Instead we should think about >how we define the MPI state after a process failure so the application >can continue running. And mechanisms for allowing the application to >replace processes. In the meeting in Chicago last week, I suggested >that we may want to look at Spawn as a way to reintroduce processes to >a job. I'm sure there are other options to explore here. > >Though I am really enjoying the conversation on the list so far with >regard to system-level C/R, I think we should not forget to direct >some of our energy on immediate, broad reaching goals. I'm not >proposing that we stop conversation on C/R and MPI, but encourage them >to continue. What I am trying to suggest is that we first focus on the >foundation, and develop solid proposals there before we turn our >attention solely to the more involved and controversial proposals on >the table. > >Some foundational questions might include: > - How does an application signal to the MPI implementation that it >wishes to enable the fault tolerance features of MPI (vs. >MPI_ERRORS_RETURN or MPI_ERRORS_FATAL) > - What is the state of MPI after a process failure with FT enabled? > - What is the state of communicators? > - If we leave a 'hole' in the communicator how do we allow the >application to interact with such a sparse communicator? > - What is the state of all of the MPI interfaces when an error occurs? > - Do we want to discuss how to re-introduce a process to an already >existing communicator? > - Do we want to discuss the ability to grow and shrink a communicator? > - We need to state clearly the theoretical bounds of any proposal on >the table? (i.e. we cannot detect failure 100% of the time, but we can >get close) > >My impression is that proposals directed at these fundamental >questions are of immediate interest to the forum. Other topics such as >piggybacking, C/R, replication (to name a few) must build on these >foundational topics. So while we are trying to address these questions >I would encourage those interested in the C/R proposal to continue >discussion. This way once the foundation is established we come out of >the C/R discussion with either a solid proposal for the group, or a >list of open research questions that need to be addressed before such >a proposal is possible. > >-- Josh > >On Oct 25, 2008, at 2:03 PM, Greg Bronevetsky wrote: > >>We're trying to come up with a good set of semantics that would >>serve a variety of system-level C/R vendors. I'm sure that if we >>target the needs of EverGrid or BLCR on Linux then we will succeed. >>However, that is not our goal. Our goal is to target user-level, >>kernel-level and VMM-level C/R (or hybrids) on the full range of >>platforms that may be supported by MPI. Focusing on Linux for the >>moment, lets look at the difference between user-level C/R and VMM- >>level C/R. VMM-level C/R sees a very large fraction of the system, >>meaning that it will likely be acceptable for the MPI_Prepare call >>to pull all message state into CPU or network card memory and just >>ensure that there is not message data on the wires and the switches. >>User-level C/R is much less capable, meaning that all MPI state must >>be not just in CPU RAM, but specifically in application-accessible >>memory space and may not have any problematic kernel-level >>attributes such as pinned/unpinned status. When interfacing a given >>C/R tool with MPI these details become important and we must specify >>them explicitly. If we do not, we'll devolve into the current >>situation where each C/R tool must interface with each MPI vendor to >>get things to work. >> >>Moving to other platforms that may not have the same definitions of >>user-level/kernel-level/VMM-level as does Linux, we have a much >>deeper problem. A given type of checkpointer virtualizes the system >>at a given level of abstraction, meaning that in order to work with >>such a checkpointer MPI_Prepare must move all MPI state above that >>level. Even if we take the most conservative approach of forcing >>MPI_Prepare to move MPI state to the highest level possible, this >>still has the basic language problem of specifying what this might >>mean on platforms that don't even exist today. I just don't know how >>to do that. >> >>As such, the only plausible way that I can see for us to provide >>checkpointing support within the MPI standard is to move this >>virtualization all the way to the level of the MPI specification and >>keep MPI internals as a black box. If the application wants to >>checkpoint MPI, it must use the MPI interface to checkpoint the >>application-visible state. If it wants to checkpoint MPI datatypes, >>it can use PMPI to wrap all datatype management calls and keep track >>of them on its own. If it wants to checkpoint message state, it can >>use piggybacking and the standard MPI communication calls to do the >>appropriate logging and coordination. We may want to add extra MPI >>calls to facilitate this (piggybacking is one such example) but the >>main point of this approach is that it works at a level that is >>natural for the MPI specification and doesn't force us to define >>internal details that have so far been left unspecified by the >>standard. >> >> >>Greg Bronevetsky >>Post-Doctoral Researcher >>1028 Building 451 >>Lawrence Livermore National Lab >>(925) 424-5756 >>bronevetsky1(a)llnl.gov >> >>At 03:09 PM 10/24/2008, Supalov, Alexander wrote: >>>Thanks. I think the word "how" below is decisive. >>> >>>The definition of MPI_Init and MPI_Finalize do not say "how" >>>processes >>>are created, and still, they work. Likewise, as soon as we can define >>>the expected outcome of the proposed calls, we can offload the >>>"how" to >>>the system - in this case, the CR system. >>> >>>Now we come to the expected outcome. Imagine we guarantee that >>>there's >>>no MPI communication between the PREPARE and RESTORE calls, and no >>>messages stuck in the wire or in the buffers. What can be stored in >>>the >>>system memory covered by CR will be stored there. The rest will be >>>restored by the RESTORE call once it gets control over this memory >>>image >>>back. This may include reinitialization of the networking hardware, >>>reestablishment of connections, reopening of the files, etc. >>> >>>What other guarantees do CR people want? >>> >>>-----Original Message----- >>>From: mpi3-ft-bounces(a)lists.mpi-forum.org >>>[mailto:[email protected]] On Behalf Of Greg >>>Bronevetsky >>>Sent: Friday, October 24, 2008 11:38 PM >>>To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>>Group; >>>MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>Subject: Re: [Mpi3-ft] system-level C/R requirements >>> >>>The imprecision comes from the MPI library's interactions with the >>>C/R tool. It seems to me that each tool/MPI library combo will have >>>to define their own semantics for the two calls. The only standard >>>here would be that these functions would in fact need to be called. >>>But at this point its not really an API but a mild constraint on how >>>the real API would be used. As such, it doesn't carry much >>>information about how checkpointing is to be done. What the >>>system-level C/R people need is a set of guarantees about where MPI >>>will put its state between the two calls. I don't see a way to give >>>them such guarantees in a standardized way. This is where we get >>>stuck: we either provide a couple of calls that has little >>>informational content but rather server as placeholders for a real or >>>go fully detailed on a platform-by-platform basic. Neither approach >>>is likely to pass by the wider forum, which is why I don't know how >>>to satisfy this need within the MPI 3.0 effort. >>> >>>It looks to me like we're standardizing something that is already >>>non-standard. >>> >>>Greg Bronevetsky >>>Post-Doctoral Researcher >>>1028 Building 451 >>>Lawrence Livermore National Lab >>>(925) 424-5756 >>>bronevetsky1(a)llnl.gov >>> >>>At 02:27 PM 10/24/2008, Supalov, Alexander wrote: >>> >Thanks. Can (or should) one define semantics better than those of >>>the >>> >MPI_INIT and MPI_FINALIZE? MPI job starts after MPI_Init. The job >>>ends >>> >after MPI_Finalize. What happens before and after is almost >>>undefined. >>> >This is about all the standard specifies, and it's rather clear >>>why: it >>> >cannot prescribe the way in which processes are started, because >>>it's >>> >very system specific. CR is possibly even more system specific. >>> > >>> >Let's get back to the proposal: >>> > >>> >MPI_PREPARE_FOR_CHECKPOINT(MPI_COMM) ~ MPI_FINALIZE >>> >MPI_RESTORE_AFTER_CHECKPOINT(MPI_COMM) ~ MPI_INIT >>> > >>> >Use MPI_COMM_WORLD for global CR. Use MPI_COMM_SELF for local CR. >>> > >>> >Call the first function immediately before the checkpoint, do the >>> >checkpoint the way you like, and call the second immediately after >>>to >>> >re-enter the MPI session where you left it. >>> > >>> >What else can be added to make this more clear and more precise than >>> >MPI_INIT and MPI_FINALIZE definitions? >>> > >>> >-----Original Message----- >>> >From: mpi3-ft-bounces(a)lists.mpi-forum.org >>> >[mailto:[email protected]] On Behalf Of Greg >>> >Bronevetsky >>> >Sent: Friday, October 24, 2008 11:14 PM >>> >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>>Group; >>> >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> >Subject: Re: [Mpi3-ft] system-level C/R requirements >>> > >>> >I think that the problem for the forum will be the unclear semantics >>> >of the new calls. MPI_Init is not a good example because it has >>>clear >>> >semantics for all users of MPI but not system-level services. The >>> >difference with the quiscence calls is that we're trying to >>>provide a >>> >way to by-pass to regular MPI semantics and plug into the middle of >>> >MPI without precisely defining how the by-pass works. Precise >>> >semantics didn't matter for MPI_Init exactly because there has never >>> >been a way to look into the MPI implementation until now. The >>> >solution to this is to provide very loose semantics to the new calls >>> >but this just means that there will actually be no standard way to >>> >use the new calls, which is why I'm afraid the forum will not like >>>it. >>> > >>> >I can think of only two things that we can compare these calls to. >>> >The first is the proposed performance hint API. However, this API is >>> >just about hints and may not be a good enough analogy for the rest >>>of >>> >the forum. The other analogy is the performance profiling APIs that >>> >some MPI implementation support. These APIs allow tools to determine >>> >some statistics about internal MPI state. If that is the analogy >>>that >>> >is drawn, then it is bad for this proposal because I don't think >>>that >>> >the performance profiling API ever got much support because of the >>> >issues that we're discussing here. >>> > >>> >Greg Bronevetsky >>> >Post-Doctoral Researcher >>> >1028 Building 451 >>> >Lawrence Livermore National Lab >>> >(925) 424-5756 >>> >bronevetsky1(a)llnl.gov >>> > >>> >At 02:03 PM 10/24/2008, Supalov, Alexander wrote: >>> > >Thanks. I can't speak for the whole Forum, but my impression is >>>that >>>if >>> > >the choice will be between solving the problem of MPI and CR on >>>one >>> > >hand, and not solving it on the other hand, a reasonable proposal >>>will >>> > >go a long way toward convincing the majority, or at least moving >>>the >>> > >discussion to a still better proposal. >>> > > >>> > >As for the number of calls, this is question of ROI. We're going >>>to >>>add >>> > >200 or so fancy calls by the latest guess, while here we have >>>just 2 >>> > >that offer basic functionality of undeniable value. This should be >>> > >acceptable. >>> > >>> > >Finally, I don't know a more implementation specific call than >>> >MPI_Init. >>> > >The proposed calls live close nearby. >>> > >>> > >>> > >>> >_______________________________________________ >>> >mpi3-ft mailing list >>> >mpi3-ft(a)lists.mpi-forum.org >>> >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> > --------------------------------------------------------------------- >>> >Intel GmbH >>> >Dornacher Strasse 1 >>> >85622 Feldkirchen/Muenchen Germany >>> >Sitz der Gesellschaft: Feldkirchen bei Muenchen >>> >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>> >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >>> >VAT Registration No.: DE129385895 >>> >Citibank Frankfurt (BLZ 502 109 00) 600119052 >>> > >>> >This e-mail and any attachments may contain confidential material >>>for >>> >the sole use of the intended recipient(s). Any review or >>>distribution >>> >by others is strictly prohibited. If you are not the intended >>> >recipient, please contact the sender and delete all copies. >>> > >>> > >>> >_______________________________________________ >>> >mpi3-ft mailing list >>> >mpi3-ft(a)lists.mpi-forum.org >>> >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>>_______________________________________________ >>>mpi3-ft mailing list >>>mpi3-ft(a)lists.mpi-forum.org >>>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>--------------------------------------------------------------------- >>>Intel GmbH >>>Dornacher Strasse 1 >>>85622 Feldkirchen/Muenchen Germany >>>Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>Registergericht: Muenchen HRB 47456 Ust.-IdNr. >>>VAT Registration No.: DE129385895 >>>Citibank Frankfurt (BLZ 502 109 00) 600119052 >>> >>>This e-mail and any attachments may contain confidential material for >>>the sole use of the intended recipient(s). Any review or distribution >>>by others is strictly prohibited. If you are not the intended >>>recipient, please contact the sender and delete all copies. >>> >>> >>>_______________________________________________ >>>mpi3-ft mailing list >>>mpi3-ft(a)lists.mpi-forum.org >>>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >>_______________________________________________ >>mpi3-ft mailing list >>mpi3-ft(a)lists.mpi-forum.org >>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Greg Bronevetsky 27 Oct '08

27 Oct '08

I think that solution (b) can be done without explicit coordination help from the communication library if the only protocol we care about is sync-and-stop. You're right though, it would be easier if the communication library invoked the sequential checkpointer than the other way around. This would also take care of protocols where we need to force checkpoints. One thing that we'll need to be careful about is situations where multiple communication libraries are involved such as with hybrid MPI/OpenMP applications. In theory it is possible to use complex protocols at both levels of parallelism and we need to be careful to not make some protocol combinations impossible. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 07:18 AM 10/27/2008, Thomas Herault wrote: >Le 27 oct. 08 ï¿½ 14:59, Mike Heffner a ï¿½crit : > >>I was referring to the MPI stack supporting the PREPARE and RESTORE >>calls. Ideally we would like to take existing MPI applications and >>support C/R support without any modifications to their source. >>Similarly, if we can separate the requirements of C/R restart from >>the application design this allows the developer to focus purely on >>the problem their application is trying to solve and not on C/R >>requirements. > >As a foreword, I agree with Josh that we discussed already this in >Chicago, and we decided to first try to support the application-based >fault tolerance, and see later if we can/should >help the system-level/ transparent fault-tolerance things. > >As it has already been said in the list, transparent checkpointing >(i.e. checkpointing without modification of the application code) can >be done without an official MPI routine exposed by the MPI library. It >depends on how you stack the checkpointing mechanism and the >communication library. > >There are at least two ways to do system-level checkpointing with MPI >libraries: > >a) If the communication library is using (i.e. calling) a >checkpointing mechanism, it can implement transparent system-level >checkpointing. It will ensure the network quiescence when/if needed, >since it is handling the network communications anyway. In this case, >the checkpointing mechanism saves only one process, and it's the role >of the MPI library to build a distributed snapshot from the collection >of processes checkpoints. > >b) If, on the other hand, you want to put the checkpointing mechanism >below the MPI library (so, the MPI library does not use the >checkpointing mechanism), you cannot ensure the coherency of the >different checkpoints without help from the communication library. >Hence, you need PREPARE/RESTORE calls exposed from the MPI library to >the lower level. > >With solution a), we are simply library/checkpoint mechanism >dependent. With solution b), we require from every MPI library to >implement a not-that-easy-to-define calls pairs, and it seems to me >that the real semantics of these calls will be system dependent as >already pointed in the list. >So, my question is: what are the benefits of solution b) as compared >to solution a)? > >Bests, >Thomas > >>However, if you intend to support C/R in a way that is completely >>separate from the MPI use of the application then you will have >>scenarios that a checkpoint is initiated while the application is >>blocking on an MPI_Recv(), MPI_Barrier(), or any other blocking >>operation. Therefore, for the most value the PREPARE and RESTORE >>operations should be able to be invoked asynchronously during the >>application's MPI communication. >> >>Mike >> >>Supalov, Alexander wrote: >>>Thanks. What stack do you mean here? -----Original Message----- >>>From: mpi3-ft-bounces(a)lists.mpi-forum.org >>>[mailto:[email protected]] On Behalf Of Mike >>>Heffner >>>Sent: Saturday, October 25, 2008 2:48 AM >>>To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>Subject: Re: [Mpi3-ft] system-level C/R requirements >>>Supalov, Alexander wrote: >>>>Thanks. I think the word "how" below is decisive. >>>> >>>>The definition of MPI_Init and MPI_Finalize do not say "how" >>>>processes >>>>are created, and still, they work. Likewise, as soon as we can >>>>define >>>>the expected outcome of the proposed calls, we can offload the "how" >>>to >>>>the system - in this case, the CR system. >>>> >>>>Now we come to the expected outcome. Imagine we guarantee that >>>>there's >>>>no MPI communication between the PREPARE and RESTORE calls, and no >>>>messages stuck in the wire or in the buffers. What can be stored in >>>the >>>>system memory covered by CR will be stored there. The rest will be >>>>restored by the RESTORE call once it gets control over this memory >>>image >>>>back. This may include reinitialization of the networking hardware, >>>>reestablishment of connections, reopening of the files, etc. >>>> >>>>What other guarantees do CR people want? >>>If the stack supported these calls asynchronously during MPI >>>communication -- either from a signal handler or from a second >>>thread -- >>>then I think that definition would go a fair way towards what would >>>be required. >>>Mike >> >> >> >>_______________________________________________ >>mpi3-ft mailing list >>mpi3-ft(a)lists.mpi-forum.org >>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Greg Bronevetsky 27 Oct '08

27 Oct '08

I'm not saying that MPI would need to be responsible for C/R. However, it does to provide clear guarantees about its state to any potential C/R system. As such, we can either (i) specify such guarantees in the spec, (ii) specify a set of terms in the spec that individual MPI implementations can use to articulate their guarantees on a case-by-case basis or (iii) allow each MPI implementation and C/R system work it out one-by-one as is currently done with batch schedulers. It sounds like you're in camp (iii) or somewhere between (ii) and (iii). To me this seems to be the only tenable position because options (i) and (ii) would be too complicated to put into the spec. However, if we lean towards option (iii), I don't see what we actually need to put into the spec in order to support it. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 06:21 AM 10/27/2008, Supalov, Alexander wrote: >Thanks. I hope to have understood your approach, and if so, I tend to >disagree. In my opinion, instead of trying to specify everything and >make MPI responsible for the CR, we may just give the MPI implementation >a charter to detect the CR system involved, and do what this CR system >expects. The particular MPI implementation and the particular CR will >then figure out the boundary between themselves. This is almost like >supporting several job management systems. There's no single interface >there, but many MPIs can work with many job managers alright thru the >mpiexec or comparable means on several platforms. > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg >Bronevetsky >Sent: Saturday, October 25, 2008 8:03 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] system-level C/R requirements > >We're trying to come up with a good set of semantics that would serve >a variety of system-level C/R vendors. I'm sure that if we target the >needs of EverGrid or BLCR on Linux then we will succeed. However, >that is not our goal. Our goal is to target user-level, kernel-level >and VMM-level C/R (or hybrids) on the full range of platforms that >may be supported by MPI. Focusing on Linux for the moment, lets look >at the difference between user-level C/R and VMM-level C/R. VMM-level >C/R sees a very large fraction of the system, meaning that it will >likely be acceptable for the MPI_Prepare call to pull all message >state into CPU or network card memory and just ensure that there is >not message data on the wires and the switches. User-level C/R is >much less capable, meaning that all MPI state must be not just in CPU >RAM, but specifically in application-accessible memory space and may >not have any problematic kernel-level attributes such as >pinned/unpinned status. When interfacing a given C/R tool with MPI >these details become important and we must specify them explicitly. >If we do not, we'll devolve into the current situation where each C/R >tool must interface with each MPI vendor to get things to work. > >Moving to other platforms that may not have the same definitions of >user-level/kernel-level/VMM-level as does Linux, we have a much >deeper problem. A given type of checkpointer virtualizes the system >at a given level of abstraction, meaning that in order to work with >such a checkpointer MPI_Prepare must move all MPI state above that >level. Even if we take the most conservative approach of forcing >MPI_Prepare to move MPI state to the highest level possible, this >still has the basic language problem of specifying what this might >mean on platforms that don't even exist today. I just don't know how >to do that. > >As such, the only plausible way that I can see for us to provide >checkpointing support within the MPI standard is to move this >virtualization all the way to the level of the MPI specification and >keep MPI internals as a black box. If the application wants to >checkpoint MPI, it must use the MPI interface to checkpoint the >application-visible state. If it wants to checkpoint MPI datatypes, >it can use PMPI to wrap all datatype management calls and keep track >of them on its own. If it wants to checkpoint message state, it can >use piggybacking and the standard MPI communication calls to do the >appropriate logging and coordination. We may want to add extra MPI >calls to facilitate this (piggybacking is one such example) but the >main point of this approach is that it works at a level that is >natural for the MPI specification and doesn't force us to define >internal details that have so far been left unspecified by the standard. > > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov > >At 03:09 PM 10/24/2008, Supalov, Alexander wrote: > >Thanks. I think the word "how" below is decisive. > > > >The definition of MPI_Init and MPI_Finalize do not say "how" processes > >are created, and still, they work. Likewise, as soon as we can define > >the expected outcome of the proposed calls, we can offload the "how" to > >the system - in this case, the CR system. > > > >Now we come to the expected outcome. Imagine we guarantee that there's > >no MPI communication between the PREPARE and RESTORE calls, and no > >messages stuck in the wire or in the buffers. What can be stored in the > >system memory covered by CR will be stored there. The rest will be > >restored by the RESTORE call once it gets control over this memory >image > >back. This may include reinitialization of the networking hardware, > >reestablishment of connections, reopening of the files, etc. > > > >What other guarantees do CR people want? > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg > >Bronevetsky > >Sent: Friday, October 24, 2008 11:38 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; > >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] system-level C/R requirements > > > >The imprecision comes from the MPI library's interactions with the > >C/R tool. It seems to me that each tool/MPI library combo will have > >to define their own semantics for the two calls. The only standard > >here would be that these functions would in fact need to be called. > >But at this point its not really an API but a mild constraint on how > >the real API would be used. As such, it doesn't carry much > >information about how checkpointing is to be done. What the > >system-level C/R people need is a set of guarantees about where MPI > >will put its state between the two calls. I don't see a way to give > >them such guarantees in a standardized way. This is where we get > >stuck: we either provide a couple of calls that has little > >informational content but rather server as placeholders for a real or > >go fully detailed on a platform-by-platform basic. Neither approach > >is likely to pass by the wider forum, which is why I don't know how > >to satisfy this need within the MPI 3.0 effort. > > > >It looks to me like we're standardizing something that is already > >non-standard. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > > > >At 02:27 PM 10/24/2008, Supalov, Alexander wrote: > > >Thanks. Can (or should) one define semantics better than those of the > > >MPI_INIT and MPI_FINALIZE? MPI job starts after MPI_Init. The job >ends > > >after MPI_Finalize. What happens before and after is almost >undefined. > > >This is about all the standard specifies, and it's rather clear why: >it > > >cannot prescribe the way in which processes are started, because it's > > >very system specific. CR is possibly even more system specific. > > > > > >Let's get back to the proposal: > > > > > >MPI_PREPARE_FOR_CHECKPOINT(MPI_COMM) ~ MPI_FINALIZE > > >MPI_RESTORE_AFTER_CHECKPOINT(MPI_COMM) ~ MPI_INIT > > > > > >Use MPI_COMM_WORLD for global CR. Use MPI_COMM_SELF for local CR. > > > > > >Call the first function immediately before the checkpoint, do the > > >checkpoint the way you like, and call the second immediately after to > > >re-enter the MPI session where you left it. > > > > > >What else can be added to make this more clear and more precise than > > >MPI_INIT and MPI_FINALIZE definitions? > > > > > >-----Original Message----- > > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > > >[mailto:[email protected]] On Behalf Of Greg > > >Bronevetsky > > >Sent: Friday, October 24, 2008 11:14 PM > > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >Group; > > >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > > >Subject: Re: [Mpi3-ft] system-level C/R requirements > > > > > >I think that the problem for the forum will be the unclear semantics > > >of the new calls. MPI_Init is not a good example because it has clear > > >semantics for all users of MPI but not system-level services. The > > >difference with the quiscence calls is that we're trying to provide a > > >way to by-pass to regular MPI semantics and plug into the middle of > > >MPI without precisely defining how the by-pass works. Precise > > >semantics didn't matter for MPI_Init exactly because there has never > > >been a way to look into the MPI implementation until now. The > > >solution to this is to provide very loose semantics to the new calls > > >but this just means that there will actually be no standard way to > > >use the new calls, which is why I'm afraid the forum will not like >it. > > > > > >I can think of only two things that we can compare these calls to. > > >The first is the proposed performance hint API. However, this API is > > >just about hints and may not be a good enough analogy for the rest of > > >the forum. The other analogy is the performance profiling APIs that > > >some MPI implementation support. These APIs allow tools to determine > > >some statistics about internal MPI state. If that is the analogy that > > >is drawn, then it is bad for this proposal because I don't think that > > >the performance profiling API ever got much support because of the > > >issues that we're discussing here. > > > > > >Greg Bronevetsky > > >Post-Doctoral Researcher > > >1028 Building 451 > > >Lawrence Livermore National Lab > > >(925) 424-5756 > > >bronevetsky1(a)llnl.gov > > > > > >At 02:03 PM 10/24/2008, Supalov, Alexander wrote: > > > >Thanks. I can't speak for the whole Forum, but my impression is >that > >if > > > >the choice will be between solving the problem of MPI and CR on one > > > >hand, and not solving it on the other hand, a reasonable proposal > >will > > > >go a long way toward convincing the majority, or at least moving >the > > > >discussion to a still better proposal. > > > > > > > >As for the number of calls, this is question of ROI. We're going to > >add > > > >200 or so fancy calls by the latest guess, while here we have just >2 > > > >that offer basic functionality of undeniable value. This should be > > > >acceptable. > > > > > > >Finally, I don't know a more implementation specific call than > > >MPI_Init. > > > >The proposed calls live close nearby. > > > > > > > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >--------------------------------------------------------------------- > > >Intel GmbH > > >Dornacher Strasse 1 > > >85622 Feldkirchen/Muenchen Germany > > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > > >VAT Registration No.: DE129385895 > > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > > > >This e-mail and any attachments may contain confidential material for > > >the sole use of the intended recipient(s). Any review or distribution > > >by others is strictly prohibited. If you are not the intended > > >recipient, please contact the sender and delete all copies. > > > > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Joseph Ruscio 27 Oct '08

27 Oct '08

On Oct 27, 2008, at 7:18 AM, Thomas Herault wrote: > So, my question is: what are the benefits of solution b) as compared > to solution a)? If you go with solution b), the only thing the MPI implementation needs to worry about is PREPAREing and RESTOREing the network state. All of the checkpoint specific issues and complexity such as queuing system integration, checkpoint file storage options like redundant local storage, etc. In case a), MPI implementations need to concern themselves with these issues AND support the non-standard checkpoint invocation API's for every different desired system checkpointer. So that's a question of whether the responsibility for system-level checkpointer integration lays on the MPI implementor or the individual checkpoint implementors. Going with b) gives MPI an opportunity to minimally specify an integration mechanism with system-level checkpointer's. If the MPI implementor does not wish to worry about these classes of CP/R systems, they just don't implement the minimal set of calls. If they do want the support, they implement the calls. A single implementation of PREPARE and RESTORE would support most system-level checkpointers. For example our checkpointer that sits completely in user-land and BLCR that sits completely in the OS would have the same set of requirements. VMM level checkpointers have been suggested as being different. Wouldn't they either have the same requirements, or be completely transparent to the MPI stack i.e. snapshotting the OS, communication device, etc and allowing the protocol to sort out lost messages? cheers, Joe

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Thomas Herault 27 Oct '08

27 Oct '08

Le 27 oct. 08 ï¿½ 14:59, Mike Heffner a ï¿½crit : > I was referring to the MPI stack supporting the PREPARE and RESTORE > calls. Ideally we would like to take existing MPI applications and > support C/R support without any modifications to their source. > Similarly, if we can separate the requirements of C/R restart from > the application design this allows the developer to focus purely on > the problem their application is trying to solve and not on C/R > requirements. > As a foreword, I agree with Josh that we discussed already this in Chicago, and we decided to first try to support the application-based fault tolerance, and see later if we can/should help the system-level/ transparent fault-tolerance things. As it has already been said in the list, transparent checkpointing (i.e. checkpointing without modification of the application code) can be done without an official MPI routine exposed by the MPI library. It depends on how you stack the checkpointing mechanism and the communication library. There are at least two ways to do system-level checkpointing with MPI libraries: a) If the communication library is using (i.e. calling) a checkpointing mechanism, it can implement transparent system-level checkpointing. It will ensure the network quiescence when/if needed, since it is handling the network communications anyway. In this case, the checkpointing mechanism saves only one process, and it's the role of the MPI library to build a distributed snapshot from the collection of processes checkpoints. b) If, on the other hand, you want to put the checkpointing mechanism below the MPI library (so, the MPI library does not use the checkpointing mechanism), you cannot ensure the coherency of the different checkpoints without help from the communication library. Hence, you need PREPARE/RESTORE calls exposed from the MPI library to the lower level. With solution a), we are simply library/checkpoint mechanism dependent. With solution b), we require from every MPI library to implement a not-that-easy-to-define calls pairs, and it seems to me that the real semantics of these calls will be system dependent as already pointed in the list. So, my question is: what are the benefits of solution b) as compared to solution a)? Bests, Thomas > However, if you intend to support C/R in a way that is completely > separate from the MPI use of the application then you will have > scenarios that a checkpoint is initiated while the application is > blocking on an MPI_Recv(), MPI_Barrier(), or any other blocking > operation. Therefore, for the most value the PREPARE and RESTORE > operations should be able to be invoked asynchronously during the > application's MPI communication. > > Mike > > Supalov, Alexander wrote: >> Thanks. What stack do you mean here? -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org >> [mailto:[email protected]] On Behalf Of Mike >> Heffner >> Sent: Saturday, October 25, 2008 2:48 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: Re: [Mpi3-ft] system-level C/R requirements >> Supalov, Alexander wrote: >>> Thanks. I think the word "how" below is decisive. >>> >>> The definition of MPI_Init and MPI_Finalize do not say "how" >>> processes >>> are created, and still, they work. Likewise, as soon as we can >>> define >>> the expected outcome of the proposed calls, we can offload the "how" >> to >>> the system - in this case, the CR system. >>> >>> Now we come to the expected outcome. Imagine we guarantee that >>> there's >>> no MPI communication between the PREPARE and RESTORE calls, and no >>> messages stuck in the wire or in the buffers. What can be stored in >> the >>> system memory covered by CR will be stored there. The rest will be >>> restored by the RESTORE call once it gets control over this memory >> image >>> back. This may include reinitialization of the networking hardware, >>> reestablishment of connections, reopening of the files, etc. >>> >>> What other guarantees do CR people want? >>> >> If the stack supported these calls asynchronously during MPI >> communication -- either from a signal handler or from a second >> thread -- >> then I think that definition would go a fair way towards what would >> be required. >> Mike > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 27 Oct '08

27 Oct '08

Thanks. This would probably be comparable to supporting threads at MPI_THREAD_SINGLE and MPI_THREAD_MULTIPLE, i.e., we would have to introduce several CR support levels: I can imagine that some MPIs may not be prepared to deal with asynchronous CR. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Mike Heffner Sent: Monday, October 27, 2008 3:00 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements I was referring to the MPI stack supporting the PREPARE and RESTORE calls. Ideally we would like to take existing MPI applications and support C/R support without any modifications to their source. Similarly, if we can separate the requirements of C/R restart from the application design this allows the developer to focus purely on the problem their application is trying to solve and not on C/R requirements. However, if you intend to support C/R in a way that is completely separate from the MPI use of the application then you will have scenarios that a checkpoint is initiated while the application is blocking on an MPI_Recv(), MPI_Barrier(), or any other blocking operation. Therefore, for the most value the PREPARE and RESTORE operations should be able to be invoked asynchronously during the application's MPI communication. Mike Supalov, Alexander wrote: > Thanks. What stack do you mean here? > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Mike Heffner > Sent: Saturday, October 25, 2008 2:48 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] system-level C/R requirements > > Supalov, Alexander wrote: >> Thanks. I think the word "how" below is decisive. >> >> The definition of MPI_Init and MPI_Finalize do not say "how" processes >> are created, and still, they work. Likewise, as soon as we can define >> the expected outcome of the proposed calls, we can offload the "how" > to >> the system - in this case, the CR system. >> >> Now we come to the expected outcome. Imagine we guarantee that there's >> no MPI communication between the PREPARE and RESTORE calls, and no >> messages stuck in the wire or in the buffers. What can be stored in > the >> system memory covered by CR will be stored there. The rest will be >> restored by the RESTORE call once it gets control over this memory > image >> back. This may include reinitialization of the networking hardware, >> reestablishment of connections, reopening of the files, etc. >> >> What other guarantees do CR people want? >> > > If the stack supported these calls asynchronously during MPI > communication -- either from a signal handler or from a second thread -- > > then I think that definition would go a fair way towards what would be > required. > > > Mike > _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Mike Heffner 27 Oct '08

27 Oct '08

I was referring to the MPI stack supporting the PREPARE and RESTORE calls. Ideally we would like to take existing MPI applications and support C/R support without any modifications to their source. Similarly, if we can separate the requirements of C/R restart from the application design this allows the developer to focus purely on the problem their application is trying to solve and not on C/R requirements. However, if you intend to support C/R in a way that is completely separate from the MPI use of the application then you will have scenarios that a checkpoint is initiated while the application is blocking on an MPI_Recv(), MPI_Barrier(), or any other blocking operation. Therefore, for the most value the PREPARE and RESTORE operations should be able to be invoked asynchronously during the application's MPI communication. Mike Supalov, Alexander wrote: > Thanks. What stack do you mean here? > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Mike Heffner > Sent: Saturday, October 25, 2008 2:48 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] system-level C/R requirements > > Supalov, Alexander wrote: >> Thanks. I think the word "how" below is decisive. >> >> The definition of MPI_Init and MPI_Finalize do not say "how" processes >> are created, and still, they work. Likewise, as soon as we can define >> the expected outcome of the proposed calls, we can offload the "how" > to >> the system - in this case, the CR system. >> >> Now we come to the expected outcome. Imagine we guarantee that there's >> no MPI communication between the PREPARE and RESTORE calls, and no >> messages stuck in the wire or in the buffers. What can be stored in > the >> system memory covered by CR will be stored there. The rest will be >> restored by the RESTORE call once it gets control over this memory > image >> back. This may include reinitialization of the networking hardware, >> reestablishment of connections, reopening of the files, etc. >> >> What other guarantees do CR people want? >> > > If the stack supported these calls asynchronously during MPI > communication -- either from a signal handler or from a second thread -- > > then I think that definition would go a fair way towards what would be > required. > > > Mike >

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 27 Oct '08

27 Oct '08

Thanks. I hope to have understood your approach, and if so, I tend to disagree. In my opinion, instead of trying to specify everything and make MPI responsible for the CR, we may just give the MPI implementation a charter to detect the CR system involved, and do what this CR system expects. The particular MPI implementation and the particular CR will then figure out the boundary between themselves. This is almost like supporting several job management systems. There's no single interface there, but many MPIs can work with many job managers alright thru the mpiexec or comparable means on several platforms. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Saturday, October 25, 2008 8:03 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements We're trying to come up with a good set of semantics that would serve a variety of system-level C/R vendors. I'm sure that if we target the needs of EverGrid or BLCR on Linux then we will succeed. However, that is not our goal. Our goal is to target user-level, kernel-level and VMM-level C/R (or hybrids) on the full range of platforms that may be supported by MPI. Focusing on Linux for the moment, lets look at the difference between user-level C/R and VMM-level C/R. VMM-level C/R sees a very large fraction of the system, meaning that it will likely be acceptable for the MPI_Prepare call to pull all message state into CPU or network card memory and just ensure that there is not message data on the wires and the switches. User-level C/R is much less capable, meaning that all MPI state must be not just in CPU RAM, but specifically in application-accessible memory space and may not have any problematic kernel-level attributes such as pinned/unpinned status. When interfacing a given C/R tool with MPI these details become important and we must specify them explicitly. If we do not, we'll devolve into the current situation where each C/R tool must interface with each MPI vendor to get things to work. Moving to other platforms that may not have the same definitions of user-level/kernel-level/VMM-level as does Linux, we have a much deeper problem. A given type of checkpointer virtualizes the system at a given level of abstraction, meaning that in order to work with such a checkpointer MPI_Prepare must move all MPI state above that level. Even if we take the most conservative approach of forcing MPI_Prepare to move MPI state to the highest level possible, this still has the basic language problem of specifying what this might mean on platforms that don't even exist today. I just don't know how to do that. As such, the only plausible way that I can see for us to provide checkpointing support within the MPI standard is to move this virtualization all the way to the level of the MPI specification and keep MPI internals as a black box. If the application wants to checkpoint MPI, it must use the MPI interface to checkpoint the application-visible state. If it wants to checkpoint MPI datatypes, it can use PMPI to wrap all datatype management calls and keep track of them on its own. If it wants to checkpoint message state, it can use piggybacking and the standard MPI communication calls to do the appropriate logging and coordination. We may want to add extra MPI calls to facilitate this (piggybacking is one such example) but the main point of this approach is that it works at a level that is natural for the MPI specification and doesn't force us to define internal details that have so far been left unspecified by the standard. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 03:09 PM 10/24/2008, Supalov, Alexander wrote: >Thanks. I think the word "how" below is decisive. > >The definition of MPI_Init and MPI_Finalize do not say "how" processes >are created, and still, they work. Likewise, as soon as we can define >the expected outcome of the proposed calls, we can offload the "how" to >the system - in this case, the CR system. > >Now we come to the expected outcome. Imagine we guarantee that there's >no MPI communication between the PREPARE and RESTORE calls, and no >messages stuck in the wire or in the buffers. What can be stored in the >system memory covered by CR will be stored there. The rest will be >restored by the RESTORE call once it gets control over this memory image >back. This may include reinitialization of the networking hardware, >reestablishment of connections, reopening of the files, etc. > >What other guarantees do CR people want? > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg >Bronevetsky >Sent: Friday, October 24, 2008 11:38 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] system-level C/R requirements > >The imprecision comes from the MPI library's interactions with the >C/R tool. It seems to me that each tool/MPI library combo will have >to define their own semantics for the two calls. The only standard >here would be that these functions would in fact need to be called. >But at this point its not really an API but a mild constraint on how >the real API would be used. As such, it doesn't carry much >information about how checkpointing is to be done. What the >system-level C/R people need is a set of guarantees about where MPI >will put its state between the two calls. I don't see a way to give >them such guarantees in a standardized way. This is where we get >stuck: we either provide a couple of calls that has little >informational content but rather server as placeholders for a real or >go fully detailed on a platform-by-platform basic. Neither approach >is likely to pass by the wider forum, which is why I don't know how >to satisfy this need within the MPI 3.0 effort. > >It looks to me like we're standardizing something that is already >non-standard. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov > >At 02:27 PM 10/24/2008, Supalov, Alexander wrote: > >Thanks. Can (or should) one define semantics better than those of the > >MPI_INIT and MPI_FINALIZE? MPI job starts after MPI_Init. The job ends > >after MPI_Finalize. What happens before and after is almost undefined. > >This is about all the standard specifies, and it's rather clear why: it > >cannot prescribe the way in which processes are started, because it's > >very system specific. CR is possibly even more system specific. > > > >Let's get back to the proposal: > > > >MPI_PREPARE_FOR_CHECKPOINT(MPI_COMM) ~ MPI_FINALIZE > >MPI_RESTORE_AFTER_CHECKPOINT(MPI_COMM) ~ MPI_INIT > > > >Use MPI_COMM_WORLD for global CR. Use MPI_COMM_SELF for local CR. > > > >Call the first function immediately before the checkpoint, do the > >checkpoint the way you like, and call the second immediately after to > >re-enter the MPI session where you left it. > > > >What else can be added to make this more clear and more precise than > >MPI_INIT and MPI_FINALIZE definitions? > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg > >Bronevetsky > >Sent: Friday, October 24, 2008 11:14 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; > >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] system-level C/R requirements > > > >I think that the problem for the forum will be the unclear semantics > >of the new calls. MPI_Init is not a good example because it has clear > >semantics for all users of MPI but not system-level services. The > >difference with the quiscence calls is that we're trying to provide a > >way to by-pass to regular MPI semantics and plug into the middle of > >MPI without precisely defining how the by-pass works. Precise > >semantics didn't matter for MPI_Init exactly because there has never > >been a way to look into the MPI implementation until now. The > >solution to this is to provide very loose semantics to the new calls > >but this just means that there will actually be no standard way to > >use the new calls, which is why I'm afraid the forum will not like it. > > > >I can think of only two things that we can compare these calls to. > >The first is the proposed performance hint API. However, this API is > >just about hints and may not be a good enough analogy for the rest of > >the forum. The other analogy is the performance profiling APIs that > >some MPI implementation support. These APIs allow tools to determine > >some statistics about internal MPI state. If that is the analogy that > >is drawn, then it is bad for this proposal because I don't think that > >the performance profiling API ever got much support because of the > >issues that we're discussing here. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > > > >At 02:03 PM 10/24/2008, Supalov, Alexander wrote: > > >Thanks. I can't speak for the whole Forum, but my impression is that >if > > >the choice will be between solving the problem of MPI and CR on one > > >hand, and not solving it on the other hand, a reasonable proposal >will > > >go a long way toward convincing the majority, or at least moving the > > >discussion to a still better proposal. > > > > > >As for the number of calls, this is question of ROI. We're going to >add > > >200 or so fancy calls by the latest guess, while here we have just 2 > > >that offer basic functionality of undeniable value. This should be > > >acceptable. > > > > >Finally, I don't know a more implementation specific call than > >MPI_Init. > > >The proposed calls live close nearby. > > > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] Point-to-point Communications recovery
by Josh Hursey 27 Oct '08

27 Oct '08

On Oct 23, 2008, at 5:33 PM, Richard Graham wrote: > (I am changing title to help keep track of discussions) > > We have not yet started a specific discussion on what to do with > communications when a process fails, so now seems as good a time as > ever. We also need to discuss what happens when we restart a > process (i.e. do we call MPI_Init()), but we should do this in a > separate thread). > > Keeping the focus on a single process failure, where we have a > source and destination (failed process), I see two cases: > > ï¿½ we do not restore the failed process > ï¿½ we do restore the failed process > > The first case seems to be trivial: > The destination is gone, so nothing to do there. > Source: ï¿½flushï¿½ all communications associated the process that has > failed. I think there are a few unanswered questions for this case that make it a bit more complex (at least for me). First, we are requiring an MPI implementation to detect failures. What kind of statement do we want in the standard about the consistency of the failure detector across the remaining, assumed alive processes? If one process fails from the perspective of one process, but not another (loss of connection to the peer) do we require that the failure detector involve a global consensus protocol or may it operate locally and possibly make the wrong assessment of the state of the process. Secondly, we left the working group meeting with the case in which a process P is waiting in a MPI_Recv() from a process Q. While waiting process Q fails. How do we wake up process P to notify them of the failure of process Q without requiring an asynchronous failure detection mechanism. We should think about ways of implementing this requirement. What if process P is in a MPI_Recv on MPI_ANY_SOURCE and process Q is in the communicator, do we signal an error or continue waiting? Finally, we must address the issue of the state of the communicators in MPI once a process fails. I think we were close to a proposal motivated by the FT-MPI semantics. We were discussing leaving a 'hole' in the communicator by default. When a process tried to communicate with a failed process, the MPI implementation returns an error, and the process decides how to deal with the hole in the communicator. It may choose to throw away the communicator. Or it may choose to convert the hole to MPI_PROC_NULL, and allow the repaired communicator to be used in all MPI functions. This forces MPI collectives, for instance, to be rewritten in such a way as to work around holes in the communicator. Additionally the interaction of process failure with global operations such as collectives (or some of the communicator operations, or I/O) is a complex question that we must try to address. I think there are a number of proposals that need to be explored both in wording and in implementation before we can try to fully address anything more complex, such as process replacement (Thought I encourage exploring this concurrently as well). Since a process replacement solution will be based on the decisions made when covering this foundational case. > > > The second case: > The destination just re-initialized itï¿½s communications > Source: ï¿½flushï¿½ all outstanding traffic to the destination (part > of the repair() call, so negotiation can happen with the > destination), and reset the internal point-to-point sequence number > so that new traffic can be matched at the destination. > > This implies that layered s/w (application or middleware) would be > responsible for regenerating any lost traffic, if this is needed. > Perhaps if would make sense to provide the ability for upper layers > to register a recovery function that could be called after repair() > is done with restoring MPI internal state. > > Greg, is the piggy-back capability you are asking for intended to > help getting the application level communications reset to the state > just before failure, so that the application can continue ? > > Rich > > > On 10/23/08 11:49 AM, "Greg Bronevetsky" <bronevetsky1(a)llnl.gov> > wrote: > >> There is one caveat here that we should be aware of. There is not >> efficient way to implement this if we want the sender of a message >> to be informed that the receiver has been reset because then every >> send becomes a send-receive, which will significantly reduce >> performance. However, if we're willing to wait until the process >> receives data from the reset process either directly or via some >> dependence through other processes, then all this can be >> implemented efficiently. >> >> Also, we should keep in mind that for some protocols we need both >> piggybacking and non-blocking collectives. The latter is to avoid >> race conditions where a process has begun a blocking collective >> call but needs to be informed of something having to do with the >> communication. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> >>> If, as part of ft mpi, some piggy-back support is provided to the >>> application, >>> then i don't think this behavior would need to be implemented in the >>> mpi library. >>> >>> Howard >>> >>> Richard Graham wrote: >>>> Can someone think of a reason to have the library do this over >>>> the app ? I can see that letting the library do this will avoid >>>> potential race conditions that could arise if we let the app do >>>> this - basically out of band with respect to the expected >>>> communications traffic. >>>> >>>> Rich >>>> >>>> >>>> On 10/21/08 11:52 PM, "Thomas Herault" <herault.thomas(a)gmail.com <herault.thomas(a)gmail.htm >>>> > > wrote: >>>> >>>> >>>> >>>> Le 21 oct. 08 ï¿½ 22:06, Howard Pritchard a ï¿½crit : >>>> >>>> > Hello Rich, >>>> > >>>> > I thought it was also agreed that if process A communicates with >>>> > failed process B >>>> > which had been restarted by another process C, and this was the >>>> > first communication >>>> > from A to B since the restart of B, A would receive the >>>> equivalent >>>> > of a ECONNRESET error. >>>> > This was in the context of a case where option 5 below is not >>>> being >>>> > used by the app. >>>> > >>>> > Howard >>>> > >>>> >>>> Hello Howard, >>>> >>>> there was still some discussions about this at the end of the >>>> session. >>>> >>>> The argument is that the application could do as well as the >>>> library >>>> to enforce this detection if this is needed: when a process is >>>> launched to replace another one, it could define a new revision/ >>>> epoch/ >>>> restart number and tag each communication with this number to >>>> implement the check. If this can be done as efficiently by the >>>> application as it would be done by the library, asking the >>>> application >>>> to do it itself would help the library to avoid the additional cost >>>> (i.e. piggybacking an integer to each message) when the application >>>> does not need that functionality. >>>> >>>> It was suggested that the library could provide a generic mean to >>>> piggyback this kind of information to each message, in a way >>>> similar >>>> as what is discussed about piggyback/message logging-based fault >>>> tolerance. >>>> >>>> Thomas >>>> >>>> > Richard Graham wrote: >>>> >> >>>> >> Here is a summary of what I think that we agreed to today. >>>> Please >>>> >> correct any errors, and add what I am missing. >>>> >> >>>> >> ï¿½ We need to be able to restore MPI_COMM_WORLD (and itï¿½s >>>> >> derivatives) to a usable state when a process fails. >>>> >> ï¿½ Restoration may involve having MPI_PROC_NULL replace >>>> the lost >>>> >> process, or may replaced the lost processes with a new process >>>> >> (have not specified how this would happen) >>>> >> ï¿½ Processes communicating directly with the failed >>>> processes will >>>> >> be notified via a returned error code about the failure. >>>> >> ï¿½ When a process is notified of the failure, >>>> comm_repair() must be >>>> >> called. Comm_repair() is not a collective call, and is what >>>> will >>>> >> initiate the communicator repair associated with the failed >>>> process. >>>> >> ï¿½ If a process wants to be notified of process failure >>>> even if it >>>> >> is not communicating directly with this process, it must >>>> register >>>> >> for this notification. >>>> >> ï¿½ We donï¿½t have enough information to know how to >>>> continue with >>>> >> support for checkpoint/restart. >>>> >> ï¿½ We need to discuss what needs to do with respect to >>>> failure of >>>> >> collective communications. >>>> >> >>>> >> There are several issues that came up with respect to these, >>>> which >>>> >> will be detailed later on. >>>> >> >>>> >> Rich >>>> >> >>>> >> _______________________________________________ >>>> >> mpi3-ft mailing list >>>> >> mpi3-ft(a)lists.mpi-forum.org <mpi3-ft(a)lists.mpi-forum.htm> >>>> >> http:// <http:///> lists.mpi-forum.org/mailman/listinfo.cgi/ >>>> mpi3-ft <http:///> >>>> >> >>>> > >>>> > >>>> > -- >>>> > >>>> > Howard Pritchard >>>> > Cray Inc. >>>> > >>>> > _______________________________________________ >>>> > mpi3-ft mailing list >>>> > mpi3-ft(a)lists.mpi-forum.org <mpi3-ft(a)lists.mpi-forum.htm> >>>> > http:// <http:///> lists.mpi-forum.org/mailman/listinfo.cgi/ >>>> mpi3-ft <http:///> >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org <mpi3-ft(a)lists.mpi-forum.htm> >>>> http:// <http:///> lists.mpi-forum.org/mailman/listinfo.cgi/mpi3- >>>> ft <http:///> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> >>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >>>> http:// <http:///> >>>> >>>> lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http:///> >>>> >>> >>> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0