I think that solution (b) can be done without explicit coordination help from the communication library if the only protocol we care about is sync-and-stop. You're right though, it would be easier if the communication library invoked the sequential checkpointer than the other way around. This would also take care of protocols where we need to force checkpoints. One thing that we'll need to be careful about is situations where multiple communication libraries are involved such as with hybrid MPI/OpenMP applications. In theory it is possible to use complex protocols at both levels of parallelism and we need to be careful to not make some protocol combinations impossible. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 [email protected] At 07:18 AM 10/27/2008, Thomas Herault wrote:

Le 27 oct. 08 � 14:59, Mike Heffner a �crit :

...

I was referring to the MPI stack supporting the PREPARE and RESTORE calls. Ideally we would like to take existing MPI applications and support C/R support without any modifications to their source. Similarly, if we can separate the requirements of C/R restart from the application design this allows the developer to focus purely on the problem their application is trying to solve and not on C/R requirements.

As a foreword, I agree with Josh that we discussed already this in Chicago, and we decided to first try to support the application-based fault tolerance, and see later if we can/should help the system-level/ transparent fault-tolerance things.

As it has already been said in the list, transparent checkpointing (i.e. checkpointing without modification of the application code) can be done without an official MPI routine exposed by the MPI library. It depends on how you stack the checkpointing mechanism and the communication library.

There are at least two ways to do system-level checkpointing with MPI libraries:

a) If the communication library is using (i.e. calling) a checkpointing mechanism, it can implement transparent system-level checkpointing. It will ensure the network quiescence when/if needed, since it is handling the network communications anyway. In this case, the checkpointing mechanism saves only one process, and it's the role of the MPI library to build a distributed snapshot from the collection of processes checkpoints.

b) If, on the other hand, you want to put the checkpointing mechanism below the MPI library (so, the MPI library does not use the checkpointing mechanism), you cannot ensure the coherency of the different checkpoints without help from the communication library. Hence, you need PREPARE/RESTORE calls exposed from the MPI library to the lower level.

With solution a), we are simply library/checkpoint mechanism dependent. With solution b), we require from every MPI library to implement a not-that-easy-to-define calls pairs, and it seems to me that the real semantics of these calls will be system dependent as already pointed in the list. So, my question is: what are the benefits of solution b) as compared to solution a)?

Bests, Thomas

...

However, if you intend to support C/R in a way that is completely separate from the MPI use of the application then you will have scenarios that a checkpoint is initiated while the application is blocking on an MPI_Recv(), MPI_Barrier(), or any other blocking operation. Therefore, for the most value the PREPARE and RESTORE operations should be able to be invoked asynchronously during the application's MPI communication.

Mike

Supalov, Alexander wrote:

...

Thanks. What stack do you mean here? -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Mike Heffner Sent: Saturday, October 25, 2008 2:48 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements Supalov, Alexander wrote:

...

Thanks. I think the word "how" below is decisive.

The definition of MPI_Init and MPI_Finalize do not say "how" processes are created, and still, they work. Likewise, as soon as we can define the expected outcome of the proposed calls, we can offload the "how" to the system - in this case, the CR system.

Now we come to the expected outcome. Imagine we guarantee that there's no MPI communication between the PREPARE and RESTORE calls, and no messages stuck in the wire or in the buffers. What can be stored in the system memory covered by CR will be stored there. The rest will be restored by the RESTORE call once it gets control over this memory image back. This may include reinitialization of the networking hardware, reestablishment of connections, reopening of the files, etc.

What other guarantees do CR people want? If the stack supported these calls asynchronously during MPI communication -- either from a signal handler or from a second thread -- then I think that definition would go a fair way towards what would be required. Mike

_______________________________________________ mpi3-ft mailing list [email protected] http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

_______________________________________________ mpi3-ft mailing list [email protected] http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft