How about: When calling validate the epoch should only change when there is a new failure detected. So when switching from an inactive to active collective mode. If there is no new failure on the communicator, then, in your example, the Ibcast would complete successfully at all processes. Since without a new failure the epoch should not change on the communicator. If there is a new failure detected before or in the validate then a new 'cut' would be made (increments a global epoch on the communicator). The Ibcast in P0 would either complete successfully or with an error (depending on how the collective is implemented). The Ibcast in P1 would match after the 'cut'. So P1 would be waiting for P0 to post a new Ibcast to match. The user should be programing around such scenarios, so that collectives match correctly after a validate. So I would say that the example is correct, but unsafe without additional process failure checking logic. So only when a process failure occurs the validate forces the cancelation of outstanding collective operations, and effectively resets matching. Thoughts? -- Josh On Tue, Dec 20, 2011 at 11:16 AM, Darius Buntinas <[email protected]> wrote:

Hmm. �There was a reason we added that.

I think this had to do with separating collectives from one "epoch" from those of another, so that no one does anything like this:

P0 � � � � � � � � � � P1 ------------------- � �------------------- MPI_Ibcast() MPI_Comm_validate() � �MPI_Comm_validate() . � � � � � � � � � � �MPI_Ibcast() MPI_Wait() � � � � � � MPI_Wait()

Maybe we should say that validate completes-with-error all outstanding collective operations. �But in that case if the user spans a collective across a validate the collective may complete successfully at one process (e.g., root of bcast) and complete-with-an-error on another, but validate() would show that no processes have failed (no process failures would normally indicate that the bcast completed successfully). �I guess we can call this a user error, and just say don't do that.

-d

On Dec 20, 2011, at 9:03 AM, Josh Hursey wrote:

...

In 17.7.1 the proposal states: �"All collective communication operations initiated before the call to MPI_COMM_VALIDATE must also complete before it is called, and no collective calls may be initiated until it has completed."

Considering the case where FailHandlers are used in the 'ALL' operating mode. In this mode, a user may want to call validate in all of the FailHanlders to synchronize them. But if the FailHandler was triggered out of a collective operation over a communicator that does -not- include the failed process then the user cannot write a correct program (since they cannot cancel, and may not be able to complete the collective operation).

My suggestion is that we remove this sentence. Do folks have a problem with this?

Thanks, Josh

-- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

_______________________________________________ mpi3-ft mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

-- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey