December 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] New revision of RTS proposal
by Josh Hursey 19 Dec '11

19 Dec '11

A new version of the document is available on the following wiki page: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main I think this one is ready to go. If you have time, take a look over it and send notes to the list. I am planning on sending this out to the MPI Forum mailing list (and attaching it to the ticket) tonight or tomorrow morning. I'm working on the slides for the meeting. I hope to have a version to circulate tomorrow afternoon. Thanks, Josh Change Log:-----------ï¿½* Minor wording touch up clarificationsï¿½* 17.6: Added clarification to MPI_Comm_reenable_any_source() that is ok to pass MPI_GROUP_IGNORE.ï¿½* 17.6: Added clarification that MPI_Comm_drain will match across process failure notification. Similar to MPI_Comm_validate.ï¿½* 17.7.1: Added matching clarification to MPI_Comm_validate.ï¿½* 17.10: Clarify that MPI_Comm_disconnect can be passed a collectively inactive communicator. On Fri, Dec 16, 2011 at 3:36 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > A new version of the document is available on the following wiki page: > ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > > We have to have this document finalized by COB Monday if at all > possible. So take a look at it and send feedback to the list. > > -- Josh > > Change Log: > ----------- > ï¿½* Fix a few typos > ï¿½* 17.5.1: Update the Process Failure handler section per discussions > ï¿½* 17.6: Touch up the MPI_ANY_SOURCE wording > ï¿½* 17.6: Clarify that if MPI_Comm_reenable_any_source is called with > an intercommunicator then it returns the group of failed processes in > the remote group. > ï¿½* 17.6: Advice to implementors that MPI_Comm_drain maybe a locally > collective operation. > ï¿½* 17.6: Extend advice to users to note that MPI_FAILHANDLER_MODE_ALL > does not guarantee that the failure handler will be called the same > number of times unless they call a validation operation to synchronize > the handlers. So they need to take care when using MPI_Comm_drain() in > this operating mode. > ï¿½* 17.6.1: Add a sentence allowing MPI_ERR_IN_STATUS to be returned > from test and completion operations even if it is just the > MPI_ERR_ANY_SOURCE_DISABLED warning. > ï¿½* 17.8.2: Communicator creation must have a collectively active input > communicator, and return uniformly at all processes. > ï¿½* 17.8.2: Communicator construction operations will match across > process failure. So they match similar to MPI_Comm_validate() and not > like other collectives. > ï¿½* 17.8.3: Inter-communicator creation operations have the same > constraints as communicator creation (Previous 2 points). > ï¿½* 17.8.4: Added example section with the communicator creation loop example. > ï¿½* 17.9: Topology creation operations match semantics of communicator creation. > ï¿½* 17.10: Dynamic creation operations (spawn and friends) match > semantics of communicator creation. > ï¿½* 17.11: Window creation operations match semantics of communicator creation. > ï¿½* 17.12.2: File_open/close match semantics of communicator creation. > ï¿½* A.1.1: Added the MPI_FAILHANDLER_MODE_*'s to the appendix > > Open Discussion Items: > ---------------------- > ï¿½* 17.6: Rename MPI_Comm_reenable_any_source to *_validate_* (?) > ï¿½* 3.10 & 17.6.2 : Do these sections conflict? Should the status only > be associated with the 'source' since MPI_Recv would have returned the > status value if the operations were called separately? > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] New revision of RTS proposal
by Josh Hursey 16 Dec '11

16 Dec '11

A new version of the document is available on the following wiki page: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main We have to have this document finalized by COB Monday if at all possible. So take a look at it and send feedback to the list. -- Josh Change Log: ----------- * Fix a few typos * 17.5.1: Update the Process Failure handler section per discussions * 17.6: Touch up the MPI_ANY_SOURCE wording * 17.6: Clarify that if MPI_Comm_reenable_any_source is called with an intercommunicator then it returns the group of failed processes in the remote group. * 17.6: Advice to implementors that MPI_Comm_drain maybe a locally collective operation. * 17.6: Extend advice to users to note that MPI_FAILHANDLER_MODE_ALL does not guarantee that the failure handler will be called the same number of times unless they call a validation operation to synchronize the handlers. So they need to take care when using MPI_Comm_drain() in this operating mode. * 17.6.1: Add a sentence allowing MPI_ERR_IN_STATUS to be returned from test and completion operations even if it is just the MPI_ERR_ANY_SOURCE_DISABLED warning. * 17.8.2: Communicator creation must have a collectively active input communicator, and return uniformly at all processes. * 17.8.2: Communicator construction operations will match across process failure. So they match similar to MPI_Comm_validate() and not like other collectives. * 17.8.3: Inter-communicator creation operations have the same constraints as communicator creation (Previous 2 points). * 17.8.4: Added example section with the communicator creation loop example. * 17.9: Topology creation operations match semantics of communicator creation. * 17.10: Dynamic creation operations (spawn and friends) match semantics of communicator creation. * 17.11: Window creation operations match semantics of communicator creation. * 17.12.2: File_open/close match semantics of communicator creation. * A.1.1: Added the MPI_FAILHANDLER_MODE_*'s to the appendix Open Discussion Items: ---------------------- * 17.6: Rename MPI_Comm_reenable_any_source to *_validate_* (?) * 3.10 & 17.6.2 : Do these sections conflict? Should the status only be associated with the 'source' since MPI_Recv would have returned the status value if the operations were called separately? -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Communicator Creation Behavior
by Josh Hursey 16 Dec '11

16 Dec '11

The example is slightly incorrect. The validates should be removed, since a failure in the last iteration would make matching them difficult. So consider the following code sketch instead: ------------------------ MPI_Comm_dup(MPI_COMM_WORLD, comm[0]); for(i=1; i < max; ++i) { r = MPI_Comm_dup(comm[i-1], comm[i]); if( r != MPI_SUCCESS ) { break; } r = MPI_Allreduce(..., comm[i], ...); if( r != MPI_SUCCESS ) { break; } } ------------------------ Thinking about this a bit more last night. I would be ok with 'Option (Yes)' where the user is required to call the communicator creation operation to match across a new process failure. So it would behave like MPI_Comm_validate() with regard to matching. It seems a bit unnatural, since the user is forced to do something that it may not want to do because the library is not going to help it by providing a consistent cut. One side question to consider is that for MPI_Win_create() if we apply the synchronizing rule does that hurt the semantics desired for those operations? I know we have discussed this in the past. -- Josh On Thu, Dec 15, 2011 at 5:09 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > For the past week or so we have been struggling with a question of > desired behavior in the presence of failures for a communication > construction operation. > > Consider the following code sketch: > ------------------------ > MPI_Comm_dup(MPI_COMM_WORLD, comm[0]); > for(i=1; i < max; ++i) { > ï¿½r = MPI_Comm_dup(comm[i-1], comm[i]); > ï¿½if( r != MPI_SUCCESS ) { > ï¿½ ï¿½MPI_Comm_validate(comm[i-1]) > ï¿½ ï¿½break; > ï¿½} > > ï¿½r = MPI_Allreduce(..., comm[i], ...); > ï¿½if( r != MPI_SUCCESS ) { > ï¿½ ï¿½MPI_Comm_validate(comm[i]) > ï¿½ ï¿½break; > ï¿½} > } > ------------------------ > Is this an incorrect sketch because of the 'breaks'? (Are the > validates necessary?) > > > Assume for the moment that MPI_Comm_dup() is collective -and- > synchronizing. This allows the communication creation operation to > guarantee that the communicator is either created everywhere or > nowhere. > > If a process failure occurs some processes will be in the > MPI_Comm_dup() at iteration X while others are in the MPI_Allreduce() > at iteration (X-1). The Allreduce will complete with an error since > the communicator becomes collectively inactive, but some could have > left early after the decision (like the leader). > > So the process stack looks like: > P0 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½P1 > --------------- ï¿½ ï¿½ ï¿½ ï¿½---------------- > Dup(comm[X-1]) ï¿½ ï¿½ ï¿½ ï¿½ Dup(comm[X-1]) > MPI_Allreduce() ï¿½ ï¿½ ï¿½ ï¿½MPI_Allreduce() > Dup(comm[X]) ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ -> Error > ï¿½-> Error > > So should P1 be required to call Dup(comm[X])? > > > Option (No): Behave like a collective > ï¿½- The operation will fail (similar to the behavior of collectives), > and continue to fail until the next MPI_Comm_validate() on the input > communicator. > ï¿½- This means that communication creation operations are required to > have as input a collectively active communicator. > ï¿½- If a process failure occurs during the operation, then the > operation will complete in error. > > Option (Yes): Behave like MPI_Comm_validate() > ï¿½- The operation will block until all processes have called it > (similar to MPI_Comm_validate() collective). > ï¿½- The input communicator need not be collectively active, since we > know all alive processes will eventually call it. > ï¿½- So if your applications intention is jump out of the loop, P1 must > first 'match' the Dup(comm[X]) even if they do not intend to use the > communicator. > ï¿½- In one variation of this: If an implementation returns an error > from Dup(comm[X]) in P0 (instead of blocking), then (after the > validate) it must return an error to the matching Dup(comm[X]) in P1. > As George mentioned on the call, an implementation could do this by > associating a counter with the number of calls to this creation > function. If the counter does not match the other processes in the > communicator then return an error until it does. This loosens the > blocking restriction a bit. > > > I have a slight preference for Option (No). I think it better fits how > a developer would expect to react to a failure condition, and it is > consistent with out behavior for other collective operations. However, > it is not without its complexities. > > What do folks think? > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Communicator Creation Behavior
by Josh Hursey 15 Dec '11

15 Dec '11

For the past week or so we have been struggling with a question of desired behavior in the presence of failures for a communication construction operation. Consider the following code sketch: ------------------------ MPI_Comm_dup(MPI_COMM_WORLD, comm[0]); for(i=1; i < max; ++i) { r = MPI_Comm_dup(comm[i-1], comm[i]); if( r != MPI_SUCCESS ) { MPI_Comm_validate(comm[i-1]) break; } r = MPI_Allreduce(..., comm[i], ...); if( r != MPI_SUCCESS ) { MPI_Comm_validate(comm[i]) break; } } ------------------------ Is this an incorrect sketch because of the 'breaks'? (Are the validates necessary?) Assume for the moment that MPI_Comm_dup() is collective -and- synchronizing. This allows the communication creation operation to guarantee that the communicator is either created everywhere or nowhere. If a process failure occurs some processes will be in the MPI_Comm_dup() at iteration X while others are in the MPI_Allreduce() at iteration (X-1). The Allreduce will complete with an error since the communicator becomes collectively inactive, but some could have left early after the decision (like the leader). So the process stack looks like: P0 P1 --------------- ---------------- Dup(comm[X-1]) Dup(comm[X-1]) MPI_Allreduce() MPI_Allreduce() Dup(comm[X]) -> Error -> Error So should P1 be required to call Dup(comm[X])? Option (No): Behave like a collective - The operation will fail (similar to the behavior of collectives), and continue to fail until the next MPI_Comm_validate() on the input communicator. - This means that communication creation operations are required to have as input a collectively active communicator. - If a process failure occurs during the operation, then the operation will complete in error. Option (Yes): Behave like MPI_Comm_validate() - The operation will block until all processes have called it (similar to MPI_Comm_validate() collective). - The input communicator need not be collectively active, since we know all alive processes will eventually call it. - So if your applications intention is jump out of the loop, P1 must first 'match' the Dup(comm[X]) even if they do not intend to use the communicator. - In one variation of this: If an implementation returns an error from Dup(comm[X]) in P0 (instead of blocking), then (after the validate) it must return an error to the matching Dup(comm[X]) in P1. As George mentioned on the call, an implementation could do this by associating a counter with the number of calls to this creation function. If the counter does not match the other processes in the communicator then return an error until it does. This loosens the blocking restriction a bit. I have a slight preference for Option (No). I think it better fits how a developer would expect to react to a failure condition, and it is consistent with out behavior for other collective operations. However, it is not without its complexities. What do folks think? -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Talking Points Notes for Plenary Session
by Josh Hursey 15 Dec '11

15 Dec '11

As I mentioned on the teleconf today, I am putting together slides for the plenary session to introduce the MPI Forum members to the proposal. Attached is a talking points summary of the slides (I'm just starting on the slide layout now). The goal is to introduce the forum to the concepts in the proposal as a way to scope the first reading. If you have a chance, let me know what you think. I am planning on sending around an initial version of the slides along with the 'first reading ready' version of the document on Monday, Dec. 19. A new version of the document will be available tomorrow (Friday, Dec. 16) for initial review. Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] FT WG Teleconference Thursday @ 1 pm EDT
by Josh Hursey 14 Dec '11

14 Dec '11

We decided on the call today to meet again on tomorrow (Thursday, Dec. 15) at 1 pm EST to continue the discussion. We can use the same call in information as below. -- Josh On Tue, Dec 13, 2011 at 4:16 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Reminder that we have a teleconf tomorrow, Wednesday 12/14/2011, at 1 pm > EST. > > Date: December 14, 2011 > Time: 1 pm EST/New York > Dial-in information: 877-801-8130 > Code: 1044056 > > > Agenda: > - Continue discussion of notes from UTK meeting > http://lists.mpi-forum.org/mpi3-ft/2011/12/0922.php > Additional notes forthcoming > - Run-Through Stabilization Proposal > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > - Preparing for MPI Forum meeting > - Other items? > > > Additional Agenda Items [as time permits]: > - Run-Through Stabilization Proposal > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > - Process Recovery (Josh H.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 > - MPI Forum notes discussion (?) > http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php > - Error to Fault changes proposal (Darius B.) > http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php > - Piggybacking? (Martin S.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - Network Channel Failure > http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php > - FT Query Interface > http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php > > > Next Meetings: > * Jan. 4, 2012 (Teleconf) > * Jan. 9, 2012 (MPI Forum Meeting) > > Respond to the list if you have any additional topics that you would > like to discuss. > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] A few (more) notes from UTK about the RTS proposal
by Josh Hursey 13 Dec '11

13 Dec '11

Darius and I met again with the folks at UTK to discuss the proposal. It was another productive meeting, and below are the notes for that meeting. As you might expect, there are a few more items for discussion. Some of my notes are fuzzy. Maybe Darius and the folks from UTK can help fill them out. -- Josh 17.6: MPI_ANY_SOURCE ------------------------- Problem: We need to decide and commit to a solution. There are three options before us to deal with MPI_ANY_SOURCE (we currently specify 3.b): 1) Do nothing. 2) Complete all outstanding MPI_ANY_SOURCE receives in error 3) Complete blocking MPI_ANY_SOURCE receives in error, and return a warning from nonblocking MPI_ANY_SOURCE receives leaving them in the matching queue. And two variation on (2) and (3): a) The reporting of the error is an implicit acknowledgement of the process failure. Therefore once the error(/warning) is returned the user can post and wait on MPI_ANY_SOURCE receives without calling a separate function. b) After reporting of the error, the MPI library does not permit the user to post any new MPI_ANY_SOURCE operations until the user calls MPI_Comm_reenable_any_source(). So explicit acknowledgement of the error. (1) By doing nothing the user must detect failures and cancel/complete the receive if they detect that it should no longer be posted. The user can use a FailHandler as the detection mechanism if they want, and either Drain all messages or send themselves a message to complete the outstanding receive. (2) In this solution all outstanding MPI_ANY_SOURCE receives are completed in error. The problem with this solution was its impact on the matching order. If the user has: Proc 1 Proc 2 --------------------------- Irecv(ANY_SRC, ANY_TAG) Irecv(2, ANY_TAG) Waitall() ----- Process 3 fails ----- Send(dest=1, tag=1) Send(dest=1, tag=2) Then the messages could match incorrectly. The discussion was that this is a bad program (not erroneous though), and that if the user wants to manage process failure they should just avoid this situation. One nice thing about this solution is that it fits the [MPI_Recv = (MPI_Irecv + MPI_Wait)] semantic constraint in the standard. The bad thing, is that it messes with the matching order. Note that the second Irecv(2, ANY_TAG) is -not- removed from the matching queue. (3) In this solution we complete blocking MPI_ANY_SOURCE receives in error, and return a warning for nonblocking MPI_ANY_SOURCE receives and keep them in the matching queue. The nice thing about this solution is that it does not mess with the matching order. The bad thing is that it introduces a new concept (a warning) and breaks the [MPI_Recv = (MPI_Irecv + MPI_Wait)] semantic constraint in the standard. (a) With the implicit acknowledgement of the error, we do not need to have a MPI_Comm_reenable_any_source() operation. This option would make example in section 17.6.3 incorrect for a multithreaded application. The application would have to either lock around the MPI_Recv to prevent the posting or more that one MPI_Recv at a time, or use MPI_Irecv operations. (b) With explicit acknowledgement of the error, the user is protected until they can assess the group of failed processes and determine if waiting is the correct thing to do (per the example in section 17.6.3). Note that if the user wants the behavior of (a) then they can call the MPI_Comm_reenable_any_source() operation in the error handler. My inclination is to have either (2) or (3) with option (b). 17.6: MPI_COMM_REENABLE_ANY_SOURCE ---------------------------------- Problem: The name of the function "MPI_COMM_REENABLE_ANY_SOURCE" is misleading since posted MPI_Irecv's are left in the matching queue. Rename to MPI_COMM_VALIDATE_ANY_SOURCE(). 'validate' being used to indicate that the user is confirming that the library excluding this set of failure is acceptable and will meet their needs going forward if they post a new MPI_ANY_SOURCE receive operation. Per the mailing list, also clarify that if it is used over an intercommunicator then it returns the process failures in the remote group. 17.5.1: Process Failure Handlers ---------------------------------- Problem: Need consistent calling behavior to allow the user to use collective operations in the handler without exposing them to deadlock. Additionally we need to make sure that the FailHandler is called the same number of times, in the same order at all processes. When reading the solutions below, keep in mind the following scenario: You have 4 communicators each with their own FailHandlers registered: A = {0, 1, 2, 3, 4} B = {0, 1, 2, 3} C = {0, 1, 2, 4} D = {0, 1} Processes 3 and 4 fail at nearly the same time. Because of how the failure detector/notifier is implemented some processes find out about the process failures in different orders. Process 0 discovers that process 3 fails and enters the FailHandler stack: A->B Process 1 discovers that process 4 fails and enters the FailHandler stack: A->C Process 2 discovers that process 3 fails and enters the FailHandler stack: A->B In the FailHandler of B and C the process is calling a collective operation (e.g., MPI_Comm_validate, or MPI_Comm_drain). At this point the two processes are deadlocked. We came up with two solutions to the FailHandler problem: Both solutions have the following clause: No FailHandlers can be called during a call to a FailHandler. However, if the consistent failure set is not complete, FailHandlers will be called after completion of the current FailHandler. Each FailHandler will be called the same number of times. FailHandlers are -not- inherited during communicator creation. (1) FailHandler is called on -all- communicators with FailHandlers registered in a consistent order at all processes. No globally consistent failure information is known before the end of MPI_Comm_validate(), if the user chooses to call it. In the scenario above without using MPI_Comm_validate in the FailHandler: Process 0 discovers that process 3 fails and enters the FailHandler stack: A->B->C->D Process 1 discovers that process 4 fails and enters the FailHandler stack: A->B->C->D Process 2 discovers that process 3 fails and enters the FailHandler stack: A->B->C->D then Process 0 discovers that process 4 fails and enters the FailHandler stack: A->B->C->D Process 1 discovers that process 3 fails and enters the FailHandler stack: A->B->C->D Process 2 discovers that process 4 fails and enters the FailHandler stack: A->B->C->D It is possible that some processes find out about the both failures at the same time so the following is possible: Process 0 discovers that process 3,4 fails and enters the FailHandler stack: A->B->C->D Process 1 discovers that process 4 fails and enters the FailHandler stack: A->B->C->D Process 2 discovers that process 3 fails and enters the FailHandler stack: A->B->C->D then Process 1 discovers that process 3 fails and enters the FailHandler stack: A->B->C->D Process 2 discovers that process 4 fails and enters the FailHandler stack: A->B->C->D In the scenario at top with using MPI_Comm_validate in the FailHandler: Process 0 discovers that process 3 fails and enters the FailHandler stack: A->B->C->D Process 0 discovers the failure of 4 during the MPI_Comm_validate() Process 1 discovers that process 4 fails and enters the FailHandler stack: A->B->C->D Process 1 discovers the failure of 3 during the MPI_Comm_validate() Process 2 discovers that process 3 fails and enters the FailHandler stack: A->B->C->D Process 2 discovers the failure of 4 during the MPI_Comm_validate() So the user is in control over the synchronization of the failure set, if they need it to be. All FailHandlers must be called in order to avoid the deadlock condition mentioned in the example at top. So even though communicator D contains no failed processes, its FailHandler is called. This is not always a bad thing. If, for example, the communication on D is no longer necessary then it may want notification of process failures outside of its communication context to cancel the communication. (2) FailHandler is called only on communicators with FailHandlers registered that include the failed process in a consistent order at all processes. The calling of the FailHandler stack is over the same globally consistent set of process failures know at the time of call. So the MPI library is calling MPI_Comm_validate internally to determine the set of failures, and the subset of FailHandlers to call. So we force a synchronization at every process failure. In the scenario above the following would occur: Process 0 discovers that process 3,4 fails and enters the FailHandler stack: A->B->C Process 1 discovers that process 3,4 fails and enters the FailHandler stack: A->B->C Process 2 discovers that process 3,4 fails and enters the FailHandler stack: A->B->C (1) is what was agreed upon in the meeting on Monday since it provides the maximum flexibility to the user. The downside, if you see it that way, is that -all- communicators are notified. 17.8.2: Communicator Management ---------------------------------- Problem: Communicator creation operations should be synchronizing in the presence of process failure. Additionally, should the communicator creation operation fail if some process fails during the operation? (Sorry if this is a bit muddled, my notes are fuzzy here) Consider the scenario below: for(i = 0; i < 10; ++i) { MPI_Comm_dup(comm[i], comm[i+1]); MPI_Comm_set_failhandler(comm[i+1]); MPI_Send(comm[i+1]); ... } If a process failure happens some processes may be in MPI_Comm_dup() in iteration (i=2) while some processes are still in MPI_Send() in iteration (i=1). The FailHandler is triggered for some processes while they are in MPI_Comm_dup() while others are in MPI_Send. In the FailHandler, the user decides to break out the the loop. So it wants the MPI_Comm_dup() to fail, and those processes in MPI_Send() to not have to call the MPI_Comm_dup() to unblock it. So the MPI_Comm_dup() should not work around emerging process failures, but return an error when they first encounter a new failure. All processes must synchronize to decide to return an error consistently. So the processes that decided to fail in the MPI_Send() are required to call the MPI_Comm_dup() to 'match' the communicator creation collective with other processes so it can decide error. (We called this 'going forward to go backward' in the meeting). What the user wants is that the MPI_Comm_dup() will cancel without having to call it from the processes that are in MPI_Send(). To further complicate things, consider if the MPI_Comm_dup() is not synchronizing. For example, if it chooses the cid from a pre-decided pool of cids. Then it is possible that some processes see the failure in different iterations that may be more than one step away. If we require the MPI_Comm_dup() to synchronize, then we cannot have this type of optimization. 17.6: MPI_COMM_DRAIN ---------------------------------- I forget what the issue was here. I have the note that 'drain should be a collective local operation'. But that was before the FailHandler discussion during the meeting. Maybe the UTK folks can fill in here. -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ 1 pm EDT
by Josh Hursey 13 Dec '11

13 Dec '11

Reminder that we have a teleconf tomorrow, Wednesday 12/14/2011, at 1 pm EDT. Date: December 14, 2011 Time: 1 pm EST/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Continue discussion of notes from UTK meeting http://lists.mpi-forum.org/mpi3-ft/2011/12/0922.php Additional notes forthcoming - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main - Preparing for MPI Forum meeting - Other items? Additional Agenda Items [as time permits]: - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Piggybacking? (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Jan. 4, 2012 (Teleconf) * Jan. 9, 2012 (MPI Forum Meeting) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] reenable_anysource
by Josh Hursey 12 Dec '11

12 Dec '11

Point-to-point operations over an intercommunicator go from the caller (in the local group) to some process in the remote group. So isn't it sufficient to say that the group returned by MPI_Comm_reenable_any_source() using an intercommunicator is representative of the failures in the remote group? We do a similar trick for MPI_Comm_validate(). So the failures in the local group in relation to the point-to-point operations for an intercommunicator do not matter when reenabling ANY_SOURCE receives since two processes in the local group cannot use that intercommunicator to communicate with one another. If the user needs to know the failures in the local group they can use the separate accessor function for it, MPI_Comm_group_failed(). Do you think that is sufficient functionality for the user? Good point though, we need to clarify this in the text. -- Josh On Fri, Dec 9, 2011 at 5:32 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: > > >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Darius Buntinas >> Sent: Friday, December 09, 2011 2:22 PM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: [Mpi3-ft] reenable_anysource >> >> >> We've proposed MPI_Comm_reenable_anysource that returns a group >> containing the failed processes known to the process at the time of the >> failure. >> >> Don't we need another function for intercommunicators that returns two >> groups of failed processes: one for the local group and another for the >> remote group? >> >> MPI_Comm_reenable_anysource_inter? (now we're definitely over 30 >> characters). >> > > Yes, but as long as it is < 140 characters, do we really care? I mean, we could still tweet about the function ;) > > (sorry, couldn't resist) > > How about we have the same name and have some arg ignored? > > MPI_Comm_reenable_anysource(comm, failed_local_group, failed_remote_group) > > With failed_remote_group only valid if comm is inter? Or does that violate some naming convention? > >> -d >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] reenable_anysource
by Sur, Sayantan 09 Dec '11

09 Dec '11

> -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Darius Buntinas > Sent: Friday, December 09, 2011 2:22 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] reenable_anysource > > > We've proposed MPI_Comm_reenable_anysource that returns a group > containing the failed processes known to the process at the time of the > failure. > > Don't we need another function for intercommunicators that returns two > groups of failed processes: one for the local group and another for the > remote group? > > MPI_Comm_reenable_anysource_inter? (now we're definitely over 30 > characters). > Yes, but as long as it is < 140 characters, do we really care? I mean, we could still tweet about the function ;) (sorry, couldn't resist) How about we have the same name and have some arg ignored? MPI_Comm_reenable_anysource(comm, failed_local_group, failed_remote_group) With failed_remote_group only valid if comm is inter? Or does that violate some naming convention? > -d > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0