Thanks for the feedback from everyone. It is interesting to read over the various approaches. I was attempting to see if I could distill any general rules from the examples. I like the guideline of "Don't use MPI_Comm_agreement in the non-error case", though even George's example will call MPI_Comm_agreement when there are no errors. It is clearly possible to use MPI_Comm_agreement in non-failures cases, but it still seems like a good guideline. An obvious rule is "Don't write code where one rank can possibly call MPI_Comm_shrink while another calls MPI_Comm_agreement". I'm still wrestling with whether MPI_Comm_agreement is necessary to write nontrivial FT code or if it is simply a convenience that makes writing FT code simpler. It seems to me that you can write some simple infinite code loops that error check MPI calls but do not use MPI_Comm_agreement, but before a rank makes any significant transition within the code (for example, calling MPI_Finalize or leaving a library), a call to MPI_Comm_agreement seems necessary. I'm assuming in all these cases that the application calls collectives. Anyhow, thanks again for the code/comments. As to Josh's question of can you ever call shrink or agreement in a callback, I believe that would be very difficult to do safely, especially if the application is known to make direct calls to either of those routines. Thanks, Dave From: George Bosilca <[email protected]> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <[email protected]> Date: 04/09/2012 11:41 AM Subject: Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application Sent by: [email protected] Dave, The MPI_Comm_agree is meant to be used in case of failure. It has a significant cost, large enough not to force it on users in __any__ case. Below you will find the FT version of your example. We started from the non fault tolerant version, and added what was required to make it fault tolerant. george. success = false; do { MPI_Comm_size(comm, &size); MPI_Comm_rank(comm, &rank); root = (0 == rank); do { if (root) read_some_data_from_a_file(buffer); rc = MPI_Bcast(buffer, .... ,root, comm); if( MPI_SUCCESS != rc ) { /* check only for FT type of errors */ MPI_Comm_revoke(comm); break; } done = do_computation(buffer, size); rc = MPI_Allreduce( &done, &success, ... MPI_OP_AND, comm ); if( MPI_SUCCESS != rc ) { /* check only for FT type of errors */ success = false; /* not defined if MPI_Allreduce failed */ MPI_Comm_revoke(comm); break; } } while(false == success); MPI_Comm_agree( comm, &success ); if( false == success ) { MPI_Comm_revoke(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(comm); comm = newcomm; } } while (false == success); _______________________________________________ mpi3-ft mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft