Hi Bill, I am in agreement with your suggestion to have a collective over a communicator that returns a new communicator containing ranks "alive some point during construction". It provides cleaner semantics. The example was merely trying to utilize the new MPI_Comm_create_group API that the Forum is considering. MPI_Comm_check provides a method to form global consensus in that all ranks in comm did call it. It does not imply anything about current status of comm, or even the status "just before" the call returns. During the interval before the next call to MPI_Comm_check, it is possible that many ranks fail. However, the app/lib using MPI knows the point where everyone was alive. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Sunday, January 15, 2012 2:41 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal One concern that I have with fault tolerant proposals has to do with races in the specification. This is an area where users often "just want it to work" but getting it right is tricky. In the example here, the "alive_group" is really only that at some moment shortly before "MPI_Comm_check" returns (and possibly not even that). After that, it is really the "group_of_processes_that_was_alive_at_some_point_in_the_past". Since there are sometimes correlations in failures, this could happen even if the initial failure is rare. An alternate form might be to have a routine, collective over a communicator, that returns a new communicator meeting some definition of "members were alive at some point during construction". It wouldn't guarantee you could use it, but it would have cleaner semantics. Bill On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote: I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and resists adding new "features" for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this: for(... really long time ...) { MPI_Comm_check(work_comm, &is_ok, &alive_group); if(!is_ok) { MPI_Comm_create_group(alive_group, ..., &new_comm); // re-balance workload and use new_comm in rest of computation MPI_Comm_free(work_comm); // get rid of old comm work_comm = new_comm; } else { // continue computation using work_comm // if some proc failed in this iteration, roll back work done in this iteration, go back to loop } } William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign