October 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] system-level C/R requirements
by Narasimhan, Kannan 24 Oct '08

24 Oct '08

[Changing the title to track discussion] Agreed -- specifiying an explicit list of platforms or OS or even resource specifics is not the way to go in a standard. My suggestion would be to explore if we can define abstract, higher-level resources to define a "state", and specify high-level actions. For instance, pinning/unpinning memory is very specific to RDMA, but maybe a "disconnect virtual connection" operation may abstract it. But this puts us into the realm of virtualizing MPI internal components/concepts .. Maybe there is a more elegant way ... -Kannan- -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Friday, October 24, 2008 12:46 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Summary of today's meeting >When I look at the restore requirements on MPI as described below, they >seem quit extensive. Including re-pining and opening any previous >opened communication handles. > I agree. Furthermore, its not just the length of the list but also the fact that it is very sensitive to platform-specific details. If we have any hope of providing this API, we'll need a good survey of what would be required on the full range of possible target platforms, ranging from BGL CNK/Catamount, to Windows/Unix to Symbian, even if there currently are no MPI implementations on a given possible platform. With the regular MPI spec we don't need to do anything so thorough because MPI is high-level enough that it is reasonable to assume that an implementation of some sort can be written for any platform. However, here we're talking about such low-level details that getting them wrong in the spec would mean that implementations on some platforms would actually be impossible. We could put in an explicit list of OSs that the quiescence API applies to but I don't think that'd fly with the forum. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Greg Bronevetsky 24 Oct '08

24 Oct '08

>When I look at the restore requirements on MPI as described below, >they seem quit extensive. Including re-pining and opening any >previous opened communication handles. > I agree. Furthermore, its not just the length of the list but also the fact that it is very sensitive to platform-specific details. If we have any hope of providing this API, we'll need a good survey of what would be required on the full range of possible target platforms, ranging from BGL CNK/Catamount, to Windows/Unix to Symbian, even if there currently are no MPI implementations on a given possible platform. With the regular MPI spec we don't need to do anything so thorough because MPI is high-level enough that it is reasonable to assume that an implementation of some sort can be written for any platform. However, here we're talking about such low-level details that getting them wrong in the spec would mean that implementations on some platforms would actually be impossible. We could put in an explicit list of OSs that the quiescence API applies to but I don't think that'd fly with the forum. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Erez Haba 24 Oct '08

24 Oct '08

Thanks Kannan, In the bullets list below you probably meant hat system-level C/R require some quiescence hooks (not application-level). When I look at the restore requirements on MPI as described below, they seem quit extensive. Including re-pining and opening any previous opened communication handles. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Narasimhan, Kannan Sent: Thursday, October 23, 2008 12:53 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Summary of today's meeting Some more notes from our discussion on the topic of MPI standard support for "checkpoint/restart": * We grouped C/R under two categories: application-directed and system-level. * System-level C/R can be accomplished via many techniques: intercepting every level of system stack, using virtualization techniques, etc. * Application-directed C/R will still require some quiescence hooks from the MPI layer (ex: asyncronous progression by the MPI layer). There was some discussion on this. * The MPI requirements for System-level checkpointing cannot be formulated until we get more data to define a "quiet state" I queried Mike Hefner on the sematics of freeze/unfreeze in their (Evergrid/Librato) transparent C/R approach, and here is his response : Question 1: What is ur definition of a quiet state (after the freeze call)? Do U expect the MPI to unpin memory? free resources? or just quiet the message traffic? We need to explicitly state the semantics here ... We defined it as a state that will provide a consistent state of the application across all processes. From the MPI standpoint, this would mean a state whereby all processes in the "freeze" state would be able to continue communication if a restart were invoked. In terms of particular resources, our CP/R software manages storing all application and, optionally, all MPI memory. This includes memory that has been allocated by either a malloc(3) call or a mmap/mremap call. If that memory has been pinned by the IB driver, we will store it to disk as well. We also store the primary process resources in use: IPCs, shared memory, file handles and file rollback state, etc. These memory regions and other resources are recorded after each process returns from the freeze API. Question 2: The same goes for restore. What is expected to be there, and what is expected to be supplied as the context.... On a restore all of the memory (and other resources such as IPCs, open files, etc.) will be recreated and reloaded with the state that was recorded at checkpoint time *before* the restart API is called. On the restart, it is expected that the MPI stack reinitialize the interconnect card, recreate necessary handles for fabric communication, and re-pin all previously pinned memory regions in use by the fabric's card. -Kannan- ________________________________ From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Erez Haba Sent: Wednesday, October 22, 2008 8:53 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Summary of today's meeting Thanks for capturing this. My comments inline... From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Tuesday, October 21, 2008 9:03 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Summary of today's meeting Here is a summary of what I think that we agreed to today. Please correct any errors, and add what I am missing. * We need to be able to restore MPI_COMM_WORLD (and it's derivatives) to a usable state when a process fails. [erezh] I think that we discussed this with reference to the comment that MPI is not usable once it returned an error. we need to address that in the current standard. (I think that this should be the first item on the list) [erezh] as I recall the second item on the list, is returning errors per call site (per the Error Reporting Rules proposal) [erez] as for this specific items, I think that the wording should be "repair" rather than restore (when repair is either making a "hole" in the communicator or "filling" the whole with a new process. * Restoration may involve having MPI_PROC_NULL replace the lost process, or may replaced the lost processes with a new process (have not specified how this would happen) [erezh] again I would replace "restoration" with "repair" [erezh] We said that we can use MPI_PROC_NULL for making a "hole". i.e., the communicator will not be in the error state anymore (thus you can receive from MPI_ANY_SOURCE or use a collective) however any direct communication with the "hole" rank is like using MPI_PROC_NULL. [erezh] We also said that replacing the lost process with a new one only applies to MPI_COMM_WORD. * Processes communicating directly with the failed processes will be notified via a returned error code about the failure. * When a process is notified of the failure, comm_repair() must be called. Comm_repair() is not a collective call, and is what will initiate the communicator repair associated with the failed process. [erezh] we also discussed "generation" or "revision" of a process rank to identify if a process was recycled. I think that we ended up saying that it's not really required and it's the application responsibility to identify a restored process where there might be a dependency on prev communication (with other ranks) * If a process wants to be notified of process failure even if it is not communicating directly with this process, it must register for this notification. * We don't have enough information to know how to continue with support for checkpoint/restart. [erezh] we discussed system level checkpoint/restart versus application aware checkpoint restart * We need to discuss what needs to do with respect to failure of collective communications. [erezh] we raised the issue of identifying asymmetric view of the communicator after a "hole" repair and its impact on collectives (e.g., the link between ranks 2 and 3 is broken but they can both comm. With rank 1) . Furthermore we explored some solution by adding information to the collective message(s) to identify that the communicator view is consistent. (we said that it requires further exploration) There are several issues that came up with respect to these, which will be detailed later on. Rich

1 0

Re: [Mpi3-ft] Point-to-point Communications recovery
by Greg Bronevetsky 23 Oct '08

23 Oct '08

>This implies that layered s/w (application or middleware) would be >responsible for regenerating any lost traffic, if this is >needed. Perhaps if would make sense to provide the ability for >upper layers to register a recovery function that could be called >after repair() is done with restoring MPI internal state. Would this function be called on the surviving processes? I'm not sure that is is necessary. It seems that a simpler way to go is to allow any subsequent MPI calls to block as necessary until the appropriate repairs have been completed. This way the MPI implementation can choose exactly when to do individual repairs and may in fact choose to perform the repair between the time it detects the problem and the time it informs the application. >Greg, is the piggy-back capability you are asking for intended to >help getting the application level communications reset to the state >just before failure, so that the application can continue ? Yes, exactly. Piggybacking + non-blocking everything makes it possible for middleware to coordinate checkpointing and log enough message and non-determinism information to restore any needed communication. Having a piggybacking API is necessary to ensure good performance. It is ok not to have all operations be non-blocking but if that is the case, we'll need good semantics on what happens when a process participates in a communication that involves the failed process (I'll define "good" later since I doubt the details matter now). Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

[Mpi3-ft] Point-to-point Communications recovery
by Richard Graham 23 Oct '08

23 Oct '08

(I am changing title to help keep track of discussions) We have not yet started a specific discussion on what to do with communications when a process fails, so now seems as good a time as ever. We also need to discuss what happens when we restart a process (i.e. do we call MPI_Init()), but we should do this in a separate thread). Keeping the focus on a single process failure, where we have a source and destination (failed process), I see two cases: * we do not restore the failed process * we do restore the failed process The first case seems to be trivial: The destination is gone, so nothing to do there. Source: ³flush² all communications associated the process that has failed. The second case: The destination just re-initialized it¹s communications Source: ³flush² all outstanding traffic to the destination (part of the repair() call, so negotiation can happen with the destination), and reset the internal point-to-point sequence number so that new traffic can be matched at the destination. This implies that layered s/w (application or middleware) would be responsible for regenerating any lost traffic, if this is needed. Perhaps if would make sense to provide the ability for upper layers to register a recovery function that could be called after repair() is done with restoring MPI internal state. Greg, is the piggy-back capability you are asking for intended to help getting the application level communications reset to the state just before failure, so that the application can continue ? Rich On 10/23/08 11:49 AM, "Greg Bronevetsky" <bronevetsky1(a)llnl.gov> wrote: > There is one caveat here that we should be aware of. There is not efficient > way to implement this if we want the sender of a message to be informed that > the receiver has been reset because then every send becomes a send-receive, > which will significantly reduce performance. However, if we're willing to wait > until the process receives data from the reset process either directly or via > some dependence through other processes, then all this can be implemented > efficiently. > > Also, we should keep in mind that for some protocols we need both piggybacking > and non-blocking collectives. The latter is to avoid race conditions where a > process has begun a blocking collective call but needs to be informed of > something having to do with the communication. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > >> If, as part of ft mpi, some piggy-back support is provided to the >> application, >> then i don't think this behavior would need to be implemented in the >> mpi library. >> >> Howard >> >> Richard Graham wrote: >>> Can someone think of a reason to have the library do this over the app ? I >>> can see that letting the library do this will avoid potential race >>> conditions that could arise if we let the app do this - basically out of >>> band with respect to the expected communications traffic. >>> >>> Rich >>> >>> >>> On 10/21/08 11:52 PM, "Thomas Herault" <herault.thomas(a)gmail.com >>> <herault.thomas(a)gmail.htm> > wrote: >>> >>> >>> >>> Le 21 oct. 08 à 22:06, Howard Pritchard a écrit : >>> >>>> > Hello Rich, >>>> > >>>> > I thought it was also agreed that if process A communicates with >>>> > failed process B >>>> > which had been restarted by another process C, and this was the >>>> > first communication >>>> > from A to B since the restart of B, A would receive the equivalent >>>> > of a ECONNRESET error. >>>> > This was in the context of a case where option 5 below is not being >>>> > used by the app. >>>> > >>>> > Howard >>>> > >>> >>> Hello Howard, >>> >>> there was still some discussions about this at the end of the session. >>> >>> The argument is that the application could do as well as the library >>> to enforce this detection if this is needed: when a process is >>> launched to replace another one, it could define a new revision/epoch/ >>> restart number and tag each communication with this number to >>> implement the check. If this can be done as efficiently by the >>> application as it would be done by the library, asking the application >>> to do it itself would help the library to avoid the additional cost >>> (i.e. piggybacking an integer to each message) when the application >>> does not need that functionality. >>> >>> It was suggested that the library could provide a generic mean to >>> piggyback this kind of information to each message, in a way similar >>> as what is discussed about piggyback/message logging-based fault >>> tolerance. >>> >>> Thomas >>> >>>> > Richard Graham wrote: >>>>> >> >>>>> >> Here is a summary of what I think that we agreed to today. Please >>>>> >> correct any errors, and add what I am missing. >>>>> >> >>>>> >> We need to be able to restore MPI_COMM_WORLD (and it¹s >>>>> >> derivatives) to a usable state when a process fails. >>>>> >> Restoration may involve having MPI_PROC_NULL replace the lost >>>>> >> process, or may replaced the lost processes with a new process >>>>> >> (have not specified how this would happen) >>>>> >> Processes communicating directly with the failed processes will >>>>> >> be notified via a returned error code about the failure. >>>>> >> When a process is notified of the failure, comm_repair() must be >>>>> >> called. Comm_repair() is not a collective call, and is what will >>>>> >> initiate the communicator repair associated with the failed process. >>>>> >> If a process wants to be notified of process failure even if it >>>>> >> is not communicating directly with this process, it must register >>>>> >> for this notification. >>>>> >> We don¹t have enough information to know how to continue with >>>>> >> support for checkpoint/restart. >>>>> >> We need to discuss what needs to do with respect to failure of >>>>> >> collective communications. >>>>> >> >>>>> >> There are several issues that came up with respect to these, which >>>>> >> will be detailed later on. >>>>> >> >>>>> >> Rich >>>>> >> >>>>> >> _______________________________________________ >>>>> >> mpi3-ft mailing list >>>>> >> mpi3-ft(a)lists.mpi-forum.org <mpi3-ft(a)lists.mpi-forum.htm> >>>>> >> http:// <http:///> lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> <http:///> >>>>> >> >>>> > >>>> > >>>> > -- >>>> > >>>> > Howard Pritchard >>>> > Cray Inc. >>>> > >>>> > _______________________________________________ >>>> > mpi3-ft mailing list >>>> > mpi3-ft(a)lists.mpi-forum.org <mpi3-ft(a)lists.mpi-forum.htm> >>>> > http:// <http:///> lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> <http:///> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org <mpi3-ft(a)lists.mpi-forum.htm> >>> http:// <http:///> lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> <http:///> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> >>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >>> http:// <http:///> >>> >>> lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http:///> >>> >> >>

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Greg Bronevetsky 23 Oct '08

23 Oct '08

>Even in the application-directed case, most MPI stacks would still >need an API call to inform them to "park" their state in a manner >that it can be correctly restarted. This might mean recording cached >memory registrations (not the actual memory regions, just the >handles associated with them), recording the open communicators and >recording the communication channels open amongst processes. Some >MPI stacks may find it more efficient to collate this state >information only at checkpoint time vs. maintaining it throughout >job execution. > >I would agree though that the message quiescence is more powerful >for the asynchronous and/or transparent checkpointing cases. One thing that bothers me is that such functionality can be easily abused. I can easily imagine MPI providing an API for quiescence at an MPI_Barrier. However, given that API I also easily see a user expecting it to work when processes are more loosely synchronized and not understanding why this fails in weird and creative ways. However, support for such more elaborate strategies would be fairly hard to put into MPI itself. The same, it seems, is true of the low-level quiscence scenario: it only works with the sync-and-stop checkpointing protocol and doesn't work with anything else. This is fine but it needs to be very clearly explained to potential users. Also, for application-level quiescence we'll need the application to give MPI a handle to which to save its quiesced state. This means that the FT API is now trending towards an actual checkpointing infrastructure built into MPI. We should tred carefully here since this is exactly what we said we didn't want to do when the working group got together. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Mike Heffner 23 Oct '08

23 Oct '08

Greg Bronevetsky wrote: > >>> What kind of quiscence are you thinking of? It seems to me that >>> applications will simply need to ensure that either no messages are >>> in-flight at the time of the checkpoint or that all such messages >>> have been logged appropriately by the application. >> >> In an application-directed C/R system that would be the ideal MPI >> quiescence. > > But that is something that the application is responsible for. Was there > anything that MPI would be responsible for? Even in the application-directed case, most MPI stacks would still need an API call to inform them to "park" their state in a manner that it can be correctly restarted. This might mean recording cached memory registrations (not the actual memory regions, just the handles associated with them), recording the open communicators and recording the communication channels open amongst processes. Some MPI stacks may find it more efficient to collate this state information only at checkpoint time vs. maintaining it throughout job execution. I would agree though that the message quiescence is more powerful for the asynchronous and/or transparent checkpointing cases. Mike -- Mike Heffner <mike.heffner(a)evergrid.com> Librato, Inc. Blacksburg, VA USA Voice: (540) 443-3500 #603

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Greg Bronevetsky 23 Oct '08

23 Oct '08

>If the application "ensures" that there are no messages in-flight, >then I agree with you. I was thinking of applications that use >asyncronous MPI communication in a multi-threaded env, potentially >leading to unmatched messages..... > Right, so what would you like MPI to do in that case? Are you focused on the global checkpointing case or the case where each process may checkpoint independently? Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Narasimhan, Kannan 23 Oct '08

23 Oct '08

If the application "ensures" that there are no messages in-flight, then I agree with you. I was thinking of applications that use asyncronous MPI communication in a multi-threaded env, potentially leading to unmatched messages..... -Kannan- ________________________________ From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Thursday, October 23, 2008 2:59 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Summary of today's meeting Application-directed C/R will still require some quiescence hooks from the MPI layer (ex: asyncronous progression by the MPI layer). There was some discussion on this. What kind of quiscence are you thinking of? It seems to me that applications will simply need to ensure that either no messages are in-flight at the time of the checkpoint or that all such messages have been logged appropriately by the application. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Greg Bronevetsky 23 Oct '08

23 Oct '08

>>What kind of quiscence are you thinking of? It seems to me that >>applications will simply need to ensure that either no messages are >>in-flight at the time of the checkpoint or that all such messages >>have been logged appropriately by the application. > >In an application-directed C/R system that would be the ideal MPI quiescence. But that is something that the application is responsible for. Was there anything that MPI would be responsible for? Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0