Hello Folks, Here is a distillation of the notes I had from the meeting for mpi ft at SC'08. I wrote them more in the styles of minutes then high level concept notes. Please add or correct if you see problems/inaccuracies. In attendance at the meeting were Kannan Narashimhan, Thomas Herault, Greg Bronevetsky, Erez Hara, Hideyuki Jitsumoto, Greg Koenig, Rich Graham, and Howard Pritchard, and Josh Hursey. There was an attempt to organize the discussion around the most recent version of Rich's "Point-to-Point Communications recovery" document. Rich updated the document during the course of the meeting. The meeting began with a discussion of the "Purpose" section of this document. The document listed symmetric and asymmetric cases. There were questions about whether or not 'asymmetric' made sense here - how can it imply process death? Rich said the document was not intended to cover communication/network errors, but what to do when a process was dead/unreachable. It was decided to remove the mention of 'asymmetric' from this part of the document. At this point Rich brouight up that he would like to turn this into a paper suitable to submission to 2009 ACM International Conference on Computing Frontiers in about four weeks. There was general agreement with the "Errors" section of the document. Discussion next moved to the "Error States" section. The point was repeated that a application process will only be able to determine that MPI has detected an error when making an MPI call, using a communicator that is in an error state. There was more discussion about communicators within a process being in a 'consistent' state. Greg brought up the idea of a process as a group of "local communicators", with Kannan adding the idea of "global communicator" - a notion of local view vs global view. Greg made an analogy to memory consistency models. Erez wanted to understand better what it means to have an error state associated with a communicator - bringing up the notion of virtual connections. One of the problems discussed was whether to keep going if a communicator is being used for ranks that are still okay. Thomas advocates lazy notification. Erez pulled up the VC document and discussion continued about allowing a communicator to be used even if one or more 'vcs' was 'bad', as long as the application wasn't trying to use these 'vcs'. The problem of MPI_ANY_SOURCE was brought into the discussion. Questions were raised about what to do with received messages from a 'bad' vc which had not yet been matched. There were also questions as to whether or not this model was consistent. Thomas' drew up a diagram on a white board showing where he thought we may have a consistency issue - the problem requiring at least 3 bodies to show up. The conclusions at this point were that the app would get an error if - it tries to send to a rank /vc in a communicator which is in error - tries to receive from a rank/vc in a communicator which is in error - if it posts an MPI_ANY_SOURCE on a communicator with one or more bad vcs. For the last case, if the post does match a message, the data is returned along with the error indication. The app can at this point decide to repair the communicator or just keep going. The app can continue to use MPI_ANY_SOURCE, and, if messages are available to be received, MPI will continue to return message data and the error status. Its up to the application to know its not necessary to repair the communicator however. It can ignore the error return but may eventually hang. There was some dissatisfaction with this policy. Kannan suggested making it an option the application could set, presumably at application startup. Another idea considered would be to never hang in the MPI_ANY_SOURCE case but just return an error. Discussion returned to the 'consistent state' problem again - led by Greg and Kannan. The current thinking is two types of repair policies: repair with hole, and repair with replace. Erez brought up that the notion that the later only would work with MPI_COMM_WORLD -at least according to previous discussions. Rich said that isn't sufficient. This then led to a lengthy discussion about how to rebuild communicators in the case of the repair with replace policy. A scenario was outlined for the 'restarted/respawned' process: - original process dies/goes away - another process, detecting an error when using MPI with the 'errored' communicator, respawns the process as part of its communicator repair procedure - new process starts - out-of-band the restarted process (more accurately the MPI library internally) knows about communicators that existed and somehow repairs them The last step was the main point of further discussion. The issue of "naming" of communicators came up. There has to be some way for the restarted process to link the opaque communicator handles returned by mpi to the application. This "naming" convention would allow for reconstructing communicators. To avoid deadlock and address scalability, repair is considered to be non- blocking, if not local. A diagram of the discussion was drawn on the whiteboard at this point: respawning process respawned/new process -------------------------- -------------------------------- comm repair start comm rejoin/patch start comm repair complete comm rejoin/patch complete The discussion leaned toward having the rejoin/patching of the communicator be a local operation - or at least have the behavior of being local to the respawned process. There was also discussion of how to handle legacy libraries with a concept of "shutting down" communicators being raised. This was not discussed in detail. Erez brought up a final issue before the room had to be given up to the next group. Can the respawning process continue to use the communicator to exchange messages with "good" ranks while the communicator is being repaired? Howard