October 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Summary of today's meeting
by Mike Heffner 23 Oct '08

23 Oct '08

[resend] Greg Bronevetsky wrote: >> Application-directed C/R will still require some quiescence >> hooks from the MPI layer (ex: asyncronous progression by the MPI >> layer). There was some discussion on this. > What kind of quiscence are you thinking of? It seems to me that > applications will simply need to ensure that either no messages are > in-flight at the time of the checkpoint or that all such messages have > been logged appropriately by the application. > In an application-directed C/R system that would be the ideal MPI quiescence. Mike -- Mike Heffner <mike.heffner(a)evergrid.com> Librato, Inc. Blacksburg, VA USA Voice: (540) 443-3500 #603

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Mike Heffner 23 Oct '08

23 Oct '08

Greg Bronevetsky wrote: >> Application-directed C/R will still require some quiescence >> hooks from the MPI layer (ex: asyncronous progression by the MPI >> layer). There was some discussion on this. > What kind of quiscence are you thinking of? It seems to me that > applications will simply need to ensure that either no messages are > in-flight at the time of the checkpoint or that all such messages have > been logged appropriately by the application. In an application-directed C/R system that would be the ideal MPI quiescence. Mike -- Mike Heffner <mike.heffner(a)evergrid.com> Librato, Inc. Blacksburg, VA USA Voice: (540) 443-3500 #603

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Greg Bronevetsky 23 Oct '08

23 Oct '08

>Application-directed C/R will still require some quiescence hooks >from the MPI layer (ex: asyncronous progression by the MPI layer). >There was some discussion on this. What kind of quiscence are you thinking of? It seems to me that applications will simply need to ensure that either no messages are in-flight at the time of the checkpoint or that all such messages have been logged appropriately by the application. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Narasimhan, Kannan 23 Oct '08

23 Oct '08

Some more notes from our discussion on the topic of MPI standard support for "checkpoint/restart": * We grouped C/R under two categories: application-directed and system-level. * System-level C/R can be accomplished via many techniques: intercepting every level of system stack, using virtualization techniques, etc. * Application-directed C/R will still require some quiescence hooks from the MPI layer (ex: asyncronous progression by the MPI layer). There was some discussion on this. * The MPI requirements for System-level checkpointing cannot be formulated until we get more data to define a "quiet state" I queried Mike Hefner on the sematics of freeze/unfreeze in their (Evergrid/Librato) transparent C/R approach, and here is his response : Question 1: What is ur definition of a quiet state (after the freeze call)? Do U expect the MPI to unpin memory? free resources? or just quiet the message traffic? We need to explicitly state the semantics here ... We defined it as a state that will provide a consistent state of the application across all processes. From the MPI standpoint, this would mean a state whereby all processes in the "freeze" state would be able to continue communication if a restart were invoked. In terms of particular resources, our CP/R software manages storing all application and, optionally, all MPI memory. This includes memory that has been allocated by either a malloc(3) call or a mmap/mremap call. If that memory has been pinned by the IB driver, we will store it to disk as well. We also store the primary process resources in use: IPCs, shared memory, file handles and file rollback state, etc. These memory regions and other resources are recorded after each process returns from the freeze API. Question 2: The same goes for restore. What is expected to be there, and what is expected to be supplied as the context.... On a restore all of the memory (and other resources such as IPCs, open files, etc.) will be recreated and reloaded with the state that was recorded at checkpoint time *before* the restart API is called. On the restart, it is expected that the MPI stack reinitialize the interconnect card, recreate necessary handles for fabric communication, and re-pin all previously pinned memory regions in use by the fabric's card. -Kannan- ________________________________ From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Erez Haba Sent: Wednesday, October 22, 2008 8:53 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Summary of today's meeting Thanks for capturing this. My comments inline... From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Tuesday, October 21, 2008 9:03 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Summary of today's meeting Here is a summary of what I think that we agreed to today. Please correct any errors, and add what I am missing. * We need to be able to restore MPI_COMM_WORLD (and it's derivatives) to a usable state when a process fails. [erezh] I think that we discussed this with reference to the comment that MPI is not usable once it returned an error. we need to address that in the current standard. (I think that this should be the first item on the list) [erezh] as I recall the second item on the list, is returning errors per call site (per the Error Reporting Rules proposal) [erez] as for this specific items, I think that the wording should be "repair" rather than restore (when repair is either making a "hole" in the communicator or "filling" the whole with a new process. * Restoration may involve having MPI_PROC_NULL replace the lost process, or may replaced the lost processes with a new process (have not specified how this would happen) [erezh] again I would replace "restoration" with "repair" [erezh] We said that we can use MPI_PROC_NULL for making a "hole". i.e., the communicator will not be in the error state anymore (thus you can receive from MPI_ANY_SOURCE or use a collective) however any direct communication with the "hole" rank is like using MPI_PROC_NULL. [erezh] We also said that replacing the lost process with a new one only applies to MPI_COMM_WORD. * Processes communicating directly with the failed processes will be notified via a returned error code about the failure. * When a process is notified of the failure, comm_repair() must be called. Comm_repair() is not a collective call, and is what will initiate the communicator repair associated with the failed process. [erezh] we also discussed "generation" or "revision" of a process rank to identify if a process was recycled. I think that we ended up saying that it's not really required and it's the application responsibility to identify a restored process where there might be a dependency on prev communication (with other ranks) * If a process wants to be notified of process failure even if it is not communicating directly with this process, it must register for this notification. * We don't have enough information to know how to continue with support for checkpoint/restart. [erezh] we discussed system level checkpoint/restart versus application aware checkpoint restart * We need to discuss what needs to do with respect to failure of collective communications. [erezh] we raised the issue of identifying asymmetric view of the communicator after a "hole" repair and its impact on collectives (e.g., the link between ranks 2 and 3 is broken but they can both comm. With rank 1) . Furthermore we explored some solution by adding information to the collective message(s) to identify that the communicator view is consistent. (we said that it requires further exploration) There are several issues that came up with respect to these, which will be detailed later on. Rich

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Greg Bronevetsky 23 Oct '08

23 Oct '08

There is one caveat here that we should be aware of. There is not efficient way to implement this if we want the sender of a message to be informed that the receiver has been reset because then every send becomes a send-receive, which will significantly reduce performance. However, if we're willing to wait until the process receives data from the reset process either directly or via some dependence through other processes, then all this can be implemented efficiently. Also, we should keep in mind that for some protocols we need both piggybacking and non-blocking collectives. The latter is to avoid race conditions where a process has begun a blocking collective call but needs to be informed of something having to do with the communication. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov >If, as part of ft mpi, some piggy-back support is provided to the application, >then i don't think this behavior would need to be implemented in the >mpi library. > >Howard > >Richard Graham wrote: >>Can someone think of a reason to have the >>library do this over the app ? I can see that >>letting the library do this will avoid >>potential race conditions that could arise if >>we let the app do this - basically out of band >>with respect to the expected communications traffic. >> >>Rich >> >> >>On 10/21/08 11:52 PM, "Thomas Herault" >><<herault.thomas(a)gmail.htm>herault.thomas(a)gmail.com> wrote: >> >> >> >>Le 21 oct. 08 à 22:06, Howard Pritchard a écrit : >> >> > Hello Rich, >> > >> > I thought it was also agreed that if process A communicates with >> > failed process B >> > which had been restarted by another process C, and this was the >> > first communication >> > from A to B since the restart of B, A would receive the equivalent >> > of a ECONNRESET error. >> > This was in the context of a case where option 5 below is not being >> > used by the app. >> > >> > Howard >> > >> >>Hello Howard, >> >>there was still some discussions about this at the end of the session. >> >>The argument is that the application could do as well as the library >>to enforce this detection if this is needed: when a process is >>launched to replace another one, it could define a new revision/epoch/ >>restart number and tag each communication with this number to >>implement the check. If this can be done as efficiently by the >>application as it would be done by the library, asking the application >>to do it itself would help the library to avoid the additional cost >>(i.e. piggybacking an integer to each message) when the application >>does not need that functionality. >> >>It was suggested that the library could provide a generic mean to >>piggyback this kind of information to each message, in a way similar >>as what is discussed about piggyback/message logging-based fault >>tolerance. >> >>Thomas >> >> > Richard Graham wrote: >> >> >> >> Here is a summary of what I think that we agreed to today. Please >> >> correct any errors, and add what I am missing. >> >> >> >> We need to be able to restore MPI_COMM_WORLD (and its >> >> derivatives) to a usable state when a process fails. >> >> Restoration may involve having MPI_PROC_NULL replace the lost >> >> process, or may replaced the lost processes with a new process >> >> (have not specified how this would happen) >> >> Processes communicating directly with the failed processes will >> >> be notified via a returned error code about the failure. >> >> When a process is notified of the failure, comm_repair() must be >> >> called. Comm_repair() is not a collective call, and is what will >> >> initiate the communicator repair associated with the failed process. >> >> If a process wants to be notified of process failure even if it >> >> is not communicating directly with this process, it must register >> >> for this notification. >> >> We dont have enough information to know how to continue with >> >> support for checkpoint/restart. >> >> We need to discuss what needs to do with respect to failure of >> >> collective communications. >> >> >> >> There are several issues that came up with respect to these, which >> >> will be detailed later on. >> >> >> >> Rich >> >> >> >> _______________________________________________ >> >> mpi3-ft mailing list >> >> <mpi3-ft(a)lists.mpi-forum.htm>mpi3-ft(a)lists.mpi-forum.org >> >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> > >> > >> > -- >> > >> > Howard Pritchard >> > Cray Inc. >> > >> > _______________________________________________ >> > mpi3-ft mailing list >> > <mpi3-ft(a)lists.mpi-forum.htm>mpi3-ft(a)lists.mpi-forum.org >> > http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >>_______________________________________________ >>mpi3-ft mailing list >><mpi3-ft(a)lists.mpi-forum.htm>mpi3-ft(a)lists.mpi-forum.org >>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> >> >> >>_______________________________________________ >>mpi3-ft mailing list >><mailto:[email protected]>mpi3-ft(a)lists.mpi-forum.org >>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > >-- > >Howard Pritchard >Cray Inc. > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Howard Pritchard 23 Oct '08

23 Oct '08

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Greg Bronevetsky 23 Oct '08

23 Oct '08

>Can someone think of a reason to have the library do this over the >app ? I can see that letting the library do this will avoid >potential race conditions that could arise if we let the app do this >- basically out of band with respect to the expected communications traffic. The application can implement this functionality by having somebody write the appropriate notification protocol as a PMPI layer and just plugging it in between the application and MPI. If we have piggybacking, this layer will also be fairly efficient. The only issue may be that piggybacking over collectives is hard to define, putting a limit on how efficient such application-level protocols will be. However, the fact is that different people will want MPI to provide many more services and protocols than MPI vendors can actually implement. As such, we're much better off putting together a robust and efficient piggybacking API than asking for more services to be implemented on top of the library. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Richard Graham 23 Oct '08

23 Oct '08

Can someone think of a reason to have the library do this over the app ? I can see that letting the library do this will avoid potential race conditions that could arise if we let the app do this - basically out of band with respect to the expected communications traffic. Rich On 10/21/08 11:52 PM, "Thomas Herault" <herault.thomas(a)gmail.com> wrote: > > > Le 21 oct. 08 à 22:06, Howard Pritchard a écrit : > >> > Hello Rich, >> > >> > I thought it was also agreed that if process A communicates with >> > failed process B >> > which had been restarted by another process C, and this was the >> > first communication >> > from A to B since the restart of B, A would receive the equivalent >> > of a ECONNRESET error. >> > This was in the context of a case where option 5 below is not being >> > used by the app. >> > >> > Howard >> > > > Hello Howard, > > there was still some discussions about this at the end of the session. > > The argument is that the application could do as well as the library > to enforce this detection if this is needed: when a process is > launched to replace another one, it could define a new revision/epoch/ > restart number and tag each communication with this number to > implement the check. If this can be done as efficiently by the > application as it would be done by the library, asking the application > to do it itself would help the library to avoid the additional cost > (i.e. piggybacking an integer to each message) when the application > does not need that functionality. > > It was suggested that the library could provide a generic mean to > piggyback this kind of information to each message, in a way similar > as what is discussed about piggyback/message logging-based fault > tolerance. > > Thomas > >> > Richard Graham wrote: >>> >> >>> >> Here is a summary of what I think that we agreed to today. Please >>> >> correct any errors, and add what I am missing. >>> >> >>> >> We need to be able to restore MPI_COMM_WORLD (and it¹s >>> >> derivatives) to a usable state when a process fails. >>> >> Restoration may involve having MPI_PROC_NULL replace the lost >>> >> process, or may replaced the lost processes with a new process >>> >> (have not specified how this would happen) >>> >> Processes communicating directly with the failed processes will >>> >> be notified via a returned error code about the failure. >>> >> When a process is notified of the failure, comm_repair() must be >>> >> called. Comm_repair() is not a collective call, and is what will >>> >> initiate the communicator repair associated with the failed process. >>> >> If a process wants to be notified of process failure even if it >>> >> is not communicating directly with this process, it must register >>> >> for this notification. >>> >> We don¹t have enough information to know how to continue with >>> >> support for checkpoint/restart. >>> >> We need to discuss what needs to do with respect to failure of >>> >> collective communications. >>> >> >>> >> There are several issues that came up with respect to these, which >>> >> will be detailed later on. >>> >> >>> >> Rich >>> >> >>> >> _______________________________________________ >>> >> mpi3-ft mailing list >>> >> mpi3-ft(a)lists.mpi-forum.org >>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> > >> > >> > -- >> > >> > Howard Pritchard >> > Cray Inc. >> > >> > _______________________________________________ >> > mpi3-ft mailing list >> > mpi3-ft(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Erez Haba 22 Oct '08

22 Oct '08

Thanks for capturing this. My comments inline... From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Tuesday, October 21, 2008 9:03 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Summary of today's meeting Here is a summary of what I think that we agreed to today. Please correct any errors, and add what I am missing. * We need to be able to restore MPI_COMM_WORLD (and it's derivatives) to a usable state when a process fails. [erezh] I think that we discussed this with reference to the comment that MPI is not usable once it returned an error. we need to address that in the current standard. (I think that this should be the first item on the list) [erezh] as I recall the second item on the list, is returning errors per call site (per the Error Reporting Rules proposal) [erez] as for this specific items, I think that the wording should be "repair" rather than restore (when repair is either making a "hole" in the communicator or "filling" the whole with a new process. * Restoration may involve having MPI_PROC_NULL replace the lost process, or may replaced the lost processes with a new process (have not specified how this would happen) [erezh] again I would replace "restoration" with "repair" [erezh] We said that we can use MPI_PROC_NULL for making a "hole". i.e., the communicator will not be in the error state anymore (thus you can receive from MPI_ANY_SOURCE or use a collective) however any direct communication with the "hole" rank is like using MPI_PROC_NULL. [erezh] We also said that replacing the lost process with a new one only applies to MPI_COMM_WORD. * Processes communicating directly with the failed processes will be notified via a returned error code about the failure. * When a process is notified of the failure, comm_repair() must be called. Comm_repair() is not a collective call, and is what will initiate the communicator repair associated with the failed process. [erezh] we also discussed "generation" or "revision" of a process rank to identify if a process was recycled. I think that we ended up saying that it's not really required and it's the application responsibility to identify a restored process where there might be a dependency on prev communication (with other ranks) * If a process wants to be notified of process failure even if it is not communicating directly with this process, it must register for this notification. * We don't have enough information to know how to continue with support for checkpoint/restart. [erezh] we discussed system level checkpoint/restart versus application aware checkpoint restart * We need to discuss what needs to do with respect to failure of collective communications. [erezh] we raised the issue of identifying asymmetric view of the communicator after a "hole" repair and its impact on collectives (e.g., the link between ranks 2 and 3 is broken but they can both comm. With rank 1) . Furthermore we explored some solution by adding information to the collective message(s) to identify that the communicator view is consistent. (we said that it requires further exploration) There are several issues that came up with respect to these, which will be detailed later on. Rich

1 0

Re: [Mpi3-ft] Meeting at SC
by Erez Haba 22 Oct '08

22 Oct '08

I'm okay with any time; as for meeting location, I'll check and let you know. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Tuesday, October 21, 2008 9:07 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Meeting at SC We talked today about meeting at SC to continue making progress on the FT work. I would like to have a 3 hour chunk of time for this, if we can get people together for this long. Here are some proposed times: Tuesday anytime in the 9am - 1:30 pm time frame. Wed - before noon Thursday - Anytime but 1:30 - 3:00 pm. What is a good time for people ? Does someone have a location where we can meet ? Thanks, Rich

1 0