Last week Rich asked me to take some notes from the MPI Forum discussion on dynamic process/fault tolerance. Those notes are enclosed. --Josh ------------------- MPI Forum 3.0 Dynamic Process/Fault Tolerance Working Group Brief Notes from Jan. 15, 2008 Meeting - Chicago, IL Chapter Coordinator: Rich Graham Notes taken by Josh Hursey * May have to describe the state of a function across a failure (and possible recovery) for all MPI functions and failure scenarios * May want to consider proposing a suggestion for implementation: - i.e.: An MPI implementation does not need to do X, but if it does decide to support X then we recommend the following as a way of implementing it. * Provide support for some class (or subset) of recovery mechanisms that may be able to be built on top of an MPI implementation of the MPI 3.0 standard. * Replication techniques should be added to the (presented) list of possible recovery techniques to consider supporting. - Question: Does considering replication techniques undermine the HPC target community of the MPI standard. - MPI interface should allow for this, maybe as an opt'in functionality. * An interface to easily piggyback data on point-to-point (and maybe collective) messages would help support some high level checkpoint/ restart techniques. * Should we expose to the application the fault recovery mechanism used? (i.e., Network path failover) - As a notification once complete? - Or as a request to authorize the recovery mechanism before it executes? * Some users have expressed an interest in having an explicit checkpoint function to identify 'good' times in the code execution to checkpoint. Ideally when the checkpoint state is minimal. * Can we ask the MPI to save some internal state around a user level checkpoint operation such that on recovery MPI objects such as datatypes or communicators are automatically recreated for the user before the application resumes execution. * Areas of MPI possibly affected by fault tolerance: - Communicators such as MPI_COMM_WORLD may change size. - MPI_Init/MPI_Finalize may need to called multiple times to support some recovery techniques. - MPI I/O needs to be looked at. - MPI Topology changes that may/will occur when a process failure and possible recovery occurs. * Collective communication: - For some collectives a process can be in the collective or on either side of it, how do the semantics of the collective operation change in response to a failure/recovery operation. - Bcast, for example, may allow some processes to exit early providing some performance benefits. A fault tolerance semantic for Bcast may require a global synchronization at the end of the Bcast to ensure a two-phase commit of the effected buffers. - We could consider loosening the synchronization constraints such that the user could choose, via an MPI API, the level of confidence they require for a process to exit a collective call. - Maybe introduce the notion of epochs that can be started and ended to help in fault isolation. * Use case: How can a user choose to enable or disable fault tolerance features? - Mostly implementation specific. (link time, compile time, ...) * Could fault tolerance only apply to a subset of the MPI API for simplicity? For example, exclude collective communications or MPI I/O. * It was stated that we should express that the fault tolerance MPI specification may not be able to provide guarantees about the stability of some data segments or operations effected by a fault, but that an MPI implementation will provide a stable enough environment to allow for post recovery. - This restriction should be explicitly stated in the standard text presented. * In a failure scenario is the fault mode/error reported local or global to the communicator? * Most people agree that this working group should step forward towards API changes to help support, but maybe not provide, fault tolerance. * A mailing list will be created for this group, and will be posted to the mpi-21 listserv.