December 2011 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] reenable_anysource
by Darius Buntinas 09 Dec '11

09 Dec '11

We've proposed MPI_Comm_reenable_anysource that returns a group containing the failed processes known to the process at the time of the failure. Don't we need another function for intercommunicators that returns two groups of failed processes: one for the local group and another for the remote group? MPI_Comm_reenable_anysource_inter? (now we're definitely over 30 characters). -d

1 0

Re: [Mpi3-ft] Teleconf on Wednesday, Dec. 14 - Shifting the time
by Darius Buntinas 09 Dec '11

09 Dec '11

That's fine for me. -d On Dec 9, 2011, at 2:40 PM, Josh Hursey wrote: > We have a teleconf scheduled for Wednesday, Dec. 14 at Noon EST/New > York. However, I (and the rest of the ORNL folks) have a conflict at > that time. Would it be a problem if we shift the meeting time to 1 pm > EST/New York on that Wednesday? > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Teleconf on Wednesday, Dec. 14 - Shifting the time
by Josh Hursey 09 Dec '11

09 Dec '11

We have a teleconf scheduled for Wednesday, Dec. 14 at Noon EST/New York. However, I (and the rest of the ORNL folks) have a conflict at that time. Would it be a problem if we shift the meeting time to 1 pm EST/New York on that Wednesday? Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] MPI_Comm_reenable_any_source Alternatives
by Josh Hursey 09 Dec '11

09 Dec '11

We have a 30 character limit, and need a better name for MPI_Comm_reenable_any_source. 'reenable' was seen as misleading since some messages are left in the matching queue, and not really 'disabled.' This function is really just a protective handshake with the user so that the MPI library knows that the user has acknowledged the loss of processes. So if we keep the base MPI_Comm__any_source then we have 20 characters. Below are a list of replacement options that I came up with from the old-dusty-thesaurus. Let me know which ones you like best. 20: MPI_Comm__any_source (base) + 5: allow + 7: approve + 5: bless + 7: certify + 5: grant + 6: permit + 6: ratify +10: reactivate + 8: reenable + 8: sanction + 8: validate I kind of like the 'validate' since we are already introducing that concept as a means of confirming with the user a defined behavior. -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] New Revision of Run-Through Stabilization Proposal (Dec. 9, 2011)
by Josh Hursey 09 Dec '11

09 Dec '11

A new version of the document is available on the following wiki page: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main Dec. 9, 2011 FTWG-Process-FT-Draft-2011-12-09.pdf Change Log: ----------- * Typo fixes. * 10.5.4: Replaced "not defined" with "equivalent to that of process failure as defined in Chapter 17". * 17.5.1: Clarify that the FailHandler's are not restricted to the context of the communicator in use. The FailHandler is triggered on an MPI operation with out regard to the communication context used in the MPI operation. * 17.5.1: Note that FailHandlers will be called in the same order at all processes, but the order is determined by MPI. * 17.5.1: Lifted restrictions on what can be called in the FailHandler. Added an exception list, which just includes MPI_Finalize at the moment. * * 17.5.6: Pulled MPI_Init wording into a separate ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/307 * 17.5.6: Replaced MPI_Init wording with an Advice to Implementors about process failure. * 17.5.6: Pulled the mpiexec wording into a separate ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/306 * 17.6: Added some clarifications for MPI_ANY_SOURCE. That pending requests can match while any_source is disabled. Once matched they are free to return successfully to the application even if any_source is still disabled. * 17.6: Clarified that MPI_Comm_drain must be able to tolerate process failure, and does not need a collectively active communicator. * 17.7.1: Clarified that MPI_Comm_validate must be able to tolerate process failure, and does not need a collectively active communicator. Open Discussion Items: ---------------------- * New name for MPI_Comm_reenable_any_source() -- See forthcoming email * Should we merge MPI_Comm_group_failed() and MPI_Comm_reenable_any_source()? So that calling MPI_Comm_group_failed() also re-enables MPI_ANY_SOURCE operations/ * 3.10 & 17.6.2 : Do these sections conflict? Should the status only be associated with the 'source' since MPI_Recv would have returned the status value if the operations were called separately? * 17.6.2: The MPI_ANY_SOURCE discussion paragraph is long, and complex. We might want to think about how to simplify the wording. * 17.5.1: Should the FailHandler be set collectively, or just locally? * 17.5.1: Should the FailHandler be called only with a consistent group of failed processes? So the internal call to the FailHandler be required do call a comm_validate to reach consensus before triggering the callbacks? * 17.5.1: Things would be easier if -all- FailHandlers were triggered when a process fails. Regardless of if the associated communication object contains a failure or not. -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] A few notes from UTK about the RTS proposal
by Graham, Richard L. 07 Dec '11

07 Dec '11

On Dec 7, 2011, at 11:55 AM, Josh Hursey wrote: > On Mon, Dec 5, 2011 at 2:51 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> >> On Dec 2, 2011, at 2:00 PM, Josh Hursey wrote: >> >>> As I mentioned on Wednesday's call, Rich and I met with the folks at >>> UTK on Thursday to discuss the run-through stabilization proposal. It >>> was quite a productive meeting. Since we did not get through the whole >>> proposal, we are having another meeting on Dec. 12 from which I will >>> surely have more notes. >>> >>> Below are a few notes that I gathered from the meeting that we should >>> discuss on the call, if not on the list beforehand. >>> >>> So this is a bit long, and I am sure that I am simplifying/missing >>> details that were discussed in the meeting. UTK folks, if there are >>> any points that are not appropriately clear, pipe-up and let us know. >>> >>> Thanks, >>> Josh >>> >>> >>> >>> 17.5.6: Startup: MPI_Init >>> ------------------------- >>> Problem: This section is still unclear, and needs further clarification. >>> >>> If MPI_Init fails to initialize the library it should return an error. >>> The user cannot call MPI_Init again, so what do we expect the user to >>> do in a fault tolerant application? >> >> At the very least, it would allow the application to print an error message. There are libraries that use MPI internally, where the user writes a non-MPI program and MPI_Init is called from within a library call. I can imagine an application that tries to init a library and if it failed, it would exit gracefully, by closing files, etc. Or it may even try a different library. >> >> I realize that the MPI standard does not guarantee anything about the state of the application/environment/etc before MPI_Init is called, so we might not be able to talk precisely about what happens if MPI_Init fails anyway. > > > Yeah. I think that the rationale should be clarified a bit in this > case. One rule of thumb with a library is that 'thou shall not call > exit unless there are extraordinary circumstances." So for a library > that is calling MPI_Init, and trying to live by this rule they would > prefer to get an error message that they can return up the stack. So > even though MPI cannot be started (and maybe the calling library) this > might not be a critical event for an application that can fall back to > a different technique. > > >> >>> If MPI_Init encounters a process failure, but can initialize properly >>> then it should return success. The process failure notification should >>> be delayed until sometime after MPI_Init. >>> >>> For process recovery, we may want to clarify "the number of processes >>> that the user asked for." This way if some number of processes F >>> failed during startup, then the application has the ability to restart >>> F processes within MPI_COMM_WORLD. We may not want to address this in >>> the stabilization proposal, but is something we should discuss for the >>> recovery proposal. >>> >>> >>> 10.5.4: Connectivity >>> ------------------------- >>> Problem: The sentiment was that by using MPI_Abort to only abort the >>> processes in the specified 'comm' then we are doing dangerous thing >>> with the notion of 'connected' as defined in this section. >>> >>> Given the bulleted statement: >>> "If a process terminates without calling MPI_FINALIZE, independent >>> processes are not affected but the effect on connected processes is >>> not defined." >>> >>> We might want to clarify the 'is not defined' portion of this >>> statement. In this section there is an additional note about >>> MPI_Abort, in the proposal we are saying the MPI_Abort only terminates >>> the processes in the specified communicator (Section 8.3). >>> >>> So we might need to look at this section and clarify as necessary. >>> >> >> We haven't changed what the standard says about MPI_Abort, we're just restating it. But it looks like we need to address the "not defined" part. That also contradicts the stabilization proposal: If a process fails, we want to define what happens with the connected processes. >> >>> >>> 17.6: MPI_ANY_SOURCE >>> ------------------------- >>> Problem: The model is too complex and can be simplified. >>> >>> A blocking MPI_Recv(ANY) will return an error when a new failure >>> occurs. A nonblocking MPI_Irecv(ANY) request, during the >>> MPI_Wait/Test, will return a 'warning' (MPI_ERR_ANY_SOURCE_DISABLED) >>> and not complete when a new failure occurs. >>> >>> There was a debate if the user needs to call another function (like >>> MPI_Comm_reenable_any_source) to reactivate the ANY_SOURCE receives, >>> or if the fact that the error was returned to the user is sufficient >>> to reactivate ANY_SOURCE. The ones that returned a 'warning' error can >>> be matched/completed if while the user is handling the error a >>> matching send arrives. So this kind of just works like an interrupt. >>> >>> Additionally, once an error is returned should we remove the >>> protection that "Any new MPI_ANY_SOURCE receive operations using a >>> communicator with MPI_ANY_SOURCE receive operations disabled will not >>> be posted, and marked with an error code of the class >>> MPI_ERR_PROC_FAIL_STOP." So new MPI_Recv(ANY) would post successfully. >>> >>> The sentiment is that the user is notified of the process failure via >>> the return code from the MPI_Wait/Test, and if they do another >>> MPI_Wait/Test or post a new ANY_SOURCE receive they implicitly >>> acknowledge the failed process. To avoid a race condition where >>> another process fails between the error return and the MPI_Wait/Test, >>> the library needs to associate an 'epoch' value with the communicator >>> to make sure that the next MPI_Wait/Test returns another error. >>> >>> I like the idea of the user doing an explicit second operation, like >>> MPI_Comm_reenable_any_source, since then we know that the user is >>> aware of the failed group (and seems more MPI-like to me). However, I >>> like the more flexible semantics. So I'm a bit split on this one. >>> >> >> There's still a race with threads: A thread posts a blocking AS receive. The receive completes with an error because some process in the comm failed. Another thread posts a blocking AS receive immediately after the receive on the first thread returns. >> >> Even though, from the MPI library's point of view, the application has been "informed" of the failure when the first receive returned with an error, there wasn't enough time for the application to react to the error and prevent the second thread from posting the receive. >> >> If we feel the anysource stuff is too complicated, we can just punt on it: Process failures don't affect AS receives. The app needs to handle potential hangs itself by using nonblocking AS receives, and failure callbacks where AS receives can be cancelled (or drained). > > > That's a good point (per my earlier email in this thread). I think we > should try to capture that scenario somewhere for future reference. > >> >>> >>> 17.6: MPI_COMM_REENABLE_ANY_SOURCE >>> ---------------------------------- >>> Problem: The name of the function "MPI_COMM_REENABLE_ANY_SOURCE" is >>> misleading since posted MPI_Irecv's are left in the matching queue. >>> >>> Posted MPI_Irecv(ANY)'s are not disabled when a new failure occurs and >>> reactivated when this function is called. The 'disable' and >>> '(re)enable' parlance seems to indicate that all of the operations >>> completed, when they may have not. >>> >>> So the suggestion was that we talk in terms of >>> deactivating/reactivating ANY_SOURCE operations. Replacing this word >>> would put us close to the 30 character limit. I don't know if there is >>> better language we can use to communicate this idea or not. >>> >> >> I never really liked the name either, I just didn't have a better alternative. People had trouble with this at the plenary too. Maybe we just need to make clear that the only thing that's being disabled/re-enabled is the ability to POST new anysource receives. >> >> I'm all for a new name, though. > > Some options: (maybe?) > > 20: MPI_Comm__any_source (base) > + 8: reenable > +10: reactivate > + 5: allow > + 7: approve > + 5: bless [rich] if we take the approach suggested at the meeting in UTK, we don't need this at all. Checking on the failed group has the same outcome. Rich > > >> >>> >>> 17.6: MPI_COMM_DRAIN >>> ---------------------------------- >>> Problem: If MPI_Comm_drain is collective and meant to be called in the >>> FailHandler then we need some ordering of when the FailHandlers are >>> triggered in order to avoid deadlocks. >>> >>> This seemed to be less of a problem with MPI_Comm_drain and more of a >>> problem with FailHandler triggering (see below). >>> >>> It was mentioned that this operation might be able to be a local >>> operation by associating a another matching bit ('epoch') to the >>> communication channel. Then incrementing the epoch will exclude all >>> previous messages from matching. We don't say that the collective >>> MPI_Comm_drain is synchronizing, so I think this would be a valid >>> implementation of this operation. Something to keep in mind. >>> >>> We need to also clarify that MPI_Comm_drain is a fault tolerant >>> collective, so there is no need to call MPI_Comm_validate beforehand. >> >> OK >> >>> >>> 17.7.1: Validating Communicators >>> ---------------------------------- >>> Problem: Would like to be able to fix all of the communication >>> problems resulting from a newly failed process outside of the critical >>> path. >>> >>> The sentiment was that with the FailHandler and >>> MPI_Comm_dup/MPI_Comm_free, one should be able to eliminate the need >>> to ever call MPI_Comm_validate. So why have it in the standard? >> >> To allow other models of FT applications. I think we'd like to avoid forcing all apps to tear all communicators down and restarting for every failure. Validate allows one to continue using existing communicators with failed processes. > > > I agree. If a programmer wants to not use it, they are able to. Having > it in there helps in (what I would say) a common style of MPI > programming. > > >> >>> So in the FailHandler the user would call MPI_Comm_dup/MPI_Comm_split >>> on the communicator provided, and get a replacement that is >>> collectively active. Then they would MPI_Comm_free the old >>> communicator (maybe doing a MPI_Comm_drain to flush messages before >>> freeing it). Then replace the comm pointer with the new dup'ed >>> communicator. >>> >>> MPI_Comm_dup does not guarantee that the resulting communicator is >>> collectively active. So this might not work in the current proposal. >>> But if it did, then you might be able to get away with this trick. >> >> If you're already duping/splitting and draining, I think doing a validate shouldn't be much more overhead, especially if this is done only on an an error. > > > Maybe we should remove the restrictions on the FailHandler to allow > the user to use any MPI function in the handler. > > >> >>> >>> 17.8.2: Communicator Management >>> ---------------------------------- >>> Problem: We make no statement on whether the newly created >>> communicator is collectively active or not. >>> >>> If you pass in a collectively inactive communicator, it may be likely >>> that you get out a collectively inactive new communicator. Since we do >>> not know if all alive processes in the communicator know of the same >>> set of failed processes. >>> >>> If we required that the new communicator be collectively active, then >>> we preclude the implementation of local functioning optimizations like >>> caching cids for MPI_Comm_dup. >>> >>> If you pass in a collectively active communicator, then the user may >>> expect that the new communicator is also collectively active. If a new >>> process failure emerges during the creation operation then the >>> reporting of it may be delayed until after the creation operation has >>> completed. So pretending that the failure happened just after the >>> creation operation. >>> >>> So do we need to explicitly clarify this point? Would making an advice >>> to implementors regarding 'the newly created communicator should be >>> collectively active' help address the problem? >> >> As you pointed out, it's not useful to say that a new communicator is collectively active, since a new failure may change that. What we can say is that a communicator created from an inactive communicator is NOT collectively active. In the case of a comm_split, it's possible that a new communicator has no failed processes. In which case, I'd imagine, that we'd want it to be collectively active. > > > So I'm going back and forth on this one. I guess if we say nothing > (like we do now), what does that mean for the program developer? Do > they have to take any precaution that they would not otherwise? I > don't think so, but I need to ponder it a bit more. > > >> >>> >>> 17.5.1: Process Failure Handlers >>> ---------------------------------- >>> Problem: FailHandlers are only called if a process fails on the >>> associated communication object. If the application is using many >>> different communicators each with a FailHandler then it is likely that >>> different FailHandlers will be triggered off of different >>> communicators. If the application uses MPI_Comm_drain in the >>> operation, this can lead to deadlock and other badness. >> >> I think we should reconsider the requirement that fail handlers are only called when an MPI call is performed on the associated object. I think it complicates things and doesn't buy us much. >> >> That said, we can still have an ordering problem. >> >>> >>> So this is a tough problem. What we want is to trigger all >>> FailHandlers including the process failure (maybe all FailHandlers >>> regardless of if they include the failed process)? Additionally, we >>> need to trigger then in a consistent order across all processes so >>> that we avoid deadlock scenarios. >>> >>> It was proposed that we create something like a global partial >>> ordering of FailHandler calls. The FailHandler >>> registration/deregistration calls would be collective (so we can >>> coordinate the call ordering). The order of the calls is determined by >>> the MPI library, but the user is guaranteed that FailHandlers will be >>> called in the same order at all processes. >>> >>> There is some pretty complex hand waving associated with doing this, >>> and I would do a disservice trying to capture here since it was not my >>> idea. But that is the general problem with FailHandlers that we need >>> to find a way to work around. >>> >>> Additionally they wanted clarification that the FailHandler is >>> triggered with a globally consistent group of failed processes. So >>> this implies that the FailHandler is calling an agreement protocol >>> (like MPI_Comm_validate) behind the scenes to generate this level of >>> consistency. I would think that the MPI library would have to delay >>> reporting of additional process failures that may occur during the >>> FailHandler operation until after they are complete. >>> >> >> I think we should keep the failure handler stuff local, without coordination. If the application needs things like partial order of failure notification, the stabilization proposal should have the features that would allow the application (or third a party library) to implement that. >> >> We talked about doing the drain and dup stuff in the failure handler in order to provide FT-MPI-like behavior. In this case, when a process fails, all communicators need to be torn down, not just the ones with failed processes. The application would need to keep a list of communicators that need to be freed, and then drain/free them in the same order at every process. >> >> We could provide a drain_all convenience function that would drain/free all communicators and just leave comm_world. This could then be used for FT-MPI-style FT. > > > Let's discuss this one on the call. > > Thanks, > Josh > > >> >> -d >> >> >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> hxxp://users.nccs.gov/~jjhursey >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > hxxp://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] A few notes from UTK about the RTS proposal
by Josh Hursey 07 Dec '11

07 Dec '11

On Mon, Dec 5, 2011 at 2:51 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > On Dec 2, 2011, at 2:00 PM, Josh Hursey wrote: > >> As I mentioned on Wednesday's call, Rich and I met with the folks at >> UTK on Thursday to discuss the run-through stabilization proposal. It >> was quite a productive meeting. Since we did not get through the whole >> proposal, we are having another meeting on Dec. 12 from which I will >> surely have more notes. >> >> Below are a few notes that I gathered from the meeting that we should >> discuss on the call, if not on the list beforehand. >> >> So this is a bit long, and I am sure that I am simplifying/missing >> details that were discussed in the meeting. UTK folks, if there are >> any points that are not appropriately clear, pipe-up and let us know. >> >> Thanks, >> Josh >> >> >> >> 17.5.6: Startup: MPI_Init >> ------------------------- >> Problem: This section is still unclear, and needs further clarification. >> >> If MPI_Init fails to initialize the library it should return an error. >> The user cannot call MPI_Init again, so what do we expect the user to >> do in a fault tolerant application? > > At the very least, it would allow the application to print an error message. ï¿½There are libraries that use MPI internally, where the user writes a non-MPI program and MPI_Init is called from within a library call. ï¿½I can imagine an application that tries to init a library and if it failed, it would exit gracefully, by closing files, etc. ï¿½Or it may even try a different library. > > I realize that the MPI standard does not guarantee anything about the state of the application/environment/etc before MPI_Init is called, so we might not be able to talk precisely about what happens if MPI_Init fails anyway. Yeah. I think that the rationale should be clarified a bit in this case. One rule of thumb with a library is that 'thou shall not call exit unless there are extraordinary circumstances." So for a library that is calling MPI_Init, and trying to live by this rule they would prefer to get an error message that they can return up the stack. So even though MPI cannot be started (and maybe the calling library) this might not be a critical event for an application that can fall back to a different technique. > >> If MPI_Init encounters a process failure, but can initialize properly >> then it should return success. The process failure notification should >> be delayed until sometime after MPI_Init. >> >> For process recovery, we may want to clarify "the number of processes >> that the user asked for." This way if some number of processes F >> failed during startup, then the application has the ability to restart >> F processes within MPI_COMM_WORLD. We may not want to address this in >> the stabilization proposal, but is something we should discuss for the >> recovery proposal. >> >> >> 10.5.4: Connectivity >> ------------------------- >> Problem: The sentiment was that by using MPI_Abort to only abort the >> processes in the specified 'comm' then we are doing dangerous thing >> with the notion of 'connected' as defined in this section. >> >> Given the bulleted statement: >> "If a process terminates without calling MPI_FINALIZE, independent >> processes are not affected but the effect on connected processes is >> not defined." >> >> We might want to clarify the 'is not defined' portion of this >> statement. In this section there is an additional note about >> MPI_Abort, in the proposal we are saying the MPI_Abort only terminates >> the processes in the specified communicator (Section 8.3). >> >> So we might need to look at this section and clarify as necessary. >> > > We haven't changed what the standard says about MPI_Abort, we're just restating it. ï¿½But it looks like we need to address the "not defined" part. ï¿½That also contradicts the stabilization proposal: ï¿½If a process fails, we want to define what happens with the connected processes. > >> >> 17.6: MPI_ANY_SOURCE >> ------------------------- >> Problem: The model is too complex and can be simplified. >> >> A blocking MPI_Recv(ANY) will return an error when a new failure >> occurs. A nonblocking MPI_Irecv(ANY) request, during the >> MPI_Wait/Test, will return a 'warning' (MPI_ERR_ANY_SOURCE_DISABLED) >> and not complete when a new failure occurs. >> >> There was a debate if the user needs to call another function (like >> MPI_Comm_reenable_any_source) to reactivate the ANY_SOURCE receives, >> or if the fact that the error was returned to the user is sufficient >> to reactivate ANY_SOURCE. The ones that returned a 'warning' error can >> be matched/completed if while the user is handling the error a >> matching send arrives. So this kind of just works like an interrupt. >> >> Additionally, once an error is returned should we remove the >> protection that "Any new MPI_ANY_SOURCE receive operations using a >> communicator with MPI_ANY_SOURCE receive operations disabled will not >> be posted, and marked with an error code of the class >> MPI_ERR_PROC_FAIL_STOP." So new MPI_Recv(ANY) would post successfully. >> >> The sentiment is that the user is notified of the process failure via >> the return code from the MPI_Wait/Test, and if they do another >> MPI_Wait/Test or post a new ANY_SOURCE receive they implicitly >> acknowledge the failed process. To avoid a race condition where >> another process fails between the error return and the MPI_Wait/Test, >> the library needs to associate an 'epoch' value with the communicator >> to make sure that the next MPI_Wait/Test returns another error. >> >> I like the idea of the user doing an explicit second operation, like >> MPI_Comm_reenable_any_source, since then we know that the user is >> aware of the failed group (and seems more MPI-like to me). However, I >> like the more flexible semantics. So I'm a bit split on this one. >> > > There's still a race with threads: ï¿½A thread posts a blocking AS receive. ï¿½The receive completes with an error because some process in the comm failed. ï¿½Another thread posts a blocking AS receive immediately after the receive on the first thread returns. > > Even though, from the MPI library's point of view, the application has been "informed" of the failure when the first receive returned with an error, there wasn't enough time for the application to react to the error and prevent the second thread from posting the receive. > > If we feel the anysource stuff is too complicated, we can just punt on it: ï¿½Process failures don't affect AS receives. ï¿½The app needs to handle potential hangs itself by using nonblocking AS receives, and failure callbacks where AS receives can be cancelled (or drained). That's a good point (per my earlier email in this thread). I think we should try to capture that scenario somewhere for future reference. > >> >> 17.6: MPI_COMM_REENABLE_ANY_SOURCE >> ---------------------------------- >> Problem: The name of the function "MPI_COMM_REENABLE_ANY_SOURCE" is >> misleading since posted MPI_Irecv's are left in the matching queue. >> >> Posted MPI_Irecv(ANY)'s are not disabled when a new failure occurs and >> reactivated when this function is called. The 'disable' and >> '(re)enable' parlance seems to indicate that all of the operations >> completed, when they may have not. >> >> So the suggestion was that we talk in terms of >> deactivating/reactivating ANY_SOURCE operations. Replacing this word >> would put us close to the 30 character limit. I don't know if there is >> better language we can use to communicate this idea or not. >> > > I never really liked the name either, I just didn't have a better alternative. ï¿½People had trouble with this at the plenary too. ï¿½Maybe we just need to make clear that the only thing that's being disabled/re-enabled is the ability to POST new anysource receives. > > I'm all for a new name, though. Some options: (maybe?) 20: MPI_Comm__any_source (base) + 8: reenable +10: reactivate + 5: allow + 7: approve + 5: bless > >> >> 17.6: MPI_COMM_DRAIN >> ---------------------------------- >> Problem: If MPI_Comm_drain is collective and meant to be called in the >> FailHandler then we need some ordering of when the FailHandlers are >> triggered in order to avoid deadlocks. >> >> This seemed to be less of a problem with MPI_Comm_drain and more of a >> problem with FailHandler triggering (see below). >> >> It was mentioned that this operation might be able to be a local >> operation by associating a another matching bit ('epoch') to the >> communication channel. Then incrementing the epoch will exclude all >> previous messages from matching. We don't say that the collective >> MPI_Comm_drain is synchronizing, so I think this would be a valid >> implementation of this operation. Something to keep in mind. >> >> We need to also clarify that MPI_Comm_drain is a fault tolerant >> collective, so there is no need to call MPI_Comm_validate beforehand. > > OK > >> >> 17.7.1: Validating Communicators >> ---------------------------------- >> Problem: Would like to be able to fix all of the communication >> problems resulting from a newly failed process outside of the critical >> path. >> >> The sentiment was that with the FailHandler and >> MPI_Comm_dup/MPI_Comm_free, one should be able to eliminate the need >> to ever call MPI_Comm_validate. So why have it in the standard? > > To allow other models of FT applications. ï¿½I think we'd like to avoid forcing all apps to tear all communicators down and restarting for every failure. ï¿½Validate allows one to continue using existing communicators with failed processes. I agree. If a programmer wants to not use it, they are able to. Having it in there helps in (what I would say) a common style of MPI programming. > >> So in the FailHandler the user would call MPI_Comm_dup/MPI_Comm_split >> on the communicator provided, and get a replacement that is >> collectively active. Then they would MPI_Comm_free the old >> communicator (maybe doing a MPI_Comm_drain to flush messages before >> freeing it). Then replace the comm pointer with the new dup'ed >> communicator. >> >> MPI_Comm_dup does not guarantee that the resulting communicator is >> collectively active. So this might not work in the current proposal. >> But if it did, then you might be able to get away with this trick. > > If you're already duping/splitting and draining, I think doing a validate shouldn't be much more overhead, especially if this is done only on an an error. Maybe we should remove the restrictions on the FailHandler to allow the user to use any MPI function in the handler. > >> >> 17.8.2: Communicator Management >> ---------------------------------- >> Problem: We make no statement on whether the newly created >> communicator is collectively active or not. >> >> If you pass in a collectively inactive communicator, it may be likely >> that you get out a collectively inactive new communicator. Since we do >> not know if all alive processes in the communicator know of the same >> set of failed processes. >> >> If we required that the new communicator be collectively active, then >> we preclude the implementation of local functioning optimizations like >> caching cids for MPI_Comm_dup. >> >> If you pass in a collectively active communicator, then the user may >> expect that the new communicator is also collectively active. If a new >> process failure emerges during the creation operation then the >> reporting of it may be delayed until after the creation operation has >> completed. So pretending that the failure happened just after the >> creation operation. >> >> So do we need to explicitly clarify this point? Would making an advice >> to implementors regarding 'the newly created communicator should be >> collectively active' help address the problem? > > As you pointed out, it's not useful to say that a new communicator is collectively active, since a new failure may change that. ï¿½What we can say is that a communicator created from an inactive communicator is NOT collectively active. ï¿½In the case of a comm_split, it's possible that a new communicator has no failed processes. ï¿½In which case, I'd imagine, that we'd want it to be collectively active. So I'm going back and forth on this one. I guess if we say nothing (like we do now), what does that mean for the program developer? Do they have to take any precaution that they would not otherwise? I don't think so, but I need to ponder it a bit more. > >> >> 17.5.1: Process Failure Handlers >> ---------------------------------- >> Problem: FailHandlers are only called if a process fails on the >> associated communication object. If the application is using many >> different communicators each with a FailHandler then it is likely that >> different FailHandlers will be triggered off of different >> communicators. If the application uses MPI_Comm_drain in the >> operation, this can lead to deadlock and other badness. > > I think we should reconsider the requirement that fail handlers are only called when an MPI call is performed on the associated object. ï¿½I think it complicates things and doesn't buy us much. > > That said, we can still have an ordering problem. > >> >> So this is a tough problem. What we want is to trigger all >> FailHandlers including the process failure (maybe all FailHandlers >> regardless of if they include the failed process)? Additionally, we >> need to trigger then in a consistent order across all processes so >> that we avoid deadlock scenarios. >> >> It was proposed that we create something like a global partial >> ordering of FailHandler calls. The FailHandler >> registration/deregistration calls would be collective (so we can >> coordinate the call ordering). The order of the calls is determined by >> the MPI library, but the user is guaranteed that FailHandlers will be >> called in the same order at all processes. >> >> There is some pretty complex hand waving associated with doing this, >> and I would do a disservice trying to capture here since it was not my >> idea. But that is the general problem with FailHandlers that we need >> to find a way to work around. >> >> Additionally they wanted clarification that the FailHandler is >> triggered with a globally consistent group of failed processes. So >> this implies that the FailHandler is calling an agreement protocol >> (like MPI_Comm_validate) behind the scenes to generate this level of >> consistency. I would think that the MPI library would have to delay >> reporting of additional process failures that may occur during the >> FailHandler operation until after they are complete. >> > > I think we should keep the failure handler stuff local, without coordination. ï¿½If the application needs things like partial order of failure notification, the stabilization proposal should have the features that would allow the application (or third a party library) to implement that. > > We talked about doing the drain and dup stuff in the failure handler in order to provide FT-MPI-like behavior. ï¿½In this case, when a process fails, all communicators need to be torn down, not just the ones with failed processes. ï¿½The application would need to keep a list of communicators that need to be freed, and then drain/free them in the same order at every process. > > We could provide a drain_all convenience function that would drain/free all communicators and just leave comm_world. ï¿½This could then be used for FT-MPI-style FT. Let's discuss this one on the call. Thanks, Josh > > -d > > >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] A few notes from UTK about the RTS proposal
by Josh Hursey 07 Dec '11

07 Dec '11

(Sorry for the lag, I got pulled away by another deadline) On Mon, Dec 5, 2011 at 4:50 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > Re-reading what I wrote, I think I wasn't clear. ï¿½I was referring to the suggestion that reenable_anysource was not needed because the user was already notified of the failure when the first IRecv(ANYSOURCE) returned an error. ï¿½I don't think that's sufficient when dealing with threads (see my example). > > If I misunderstood what was suggested at the meeting, then ignore what I said. > > I do agree with Rich that both should return an error. ï¿½And any subsequent anysources should as well until reenable_anysource (or whatever we name it) gets called. > > Is there another way to deal with the deadlock? ï¿½We can, of course, punt and leave it up to the user as I suggested later. > > -d Darius: I think this is a good example of when we would need the reenable_anysource function. If I may, just to illustrate your point, consider the following scenario: Thread A Thread B --------------------------- MPI_Recv(AS) *** Process X fails **** -> Error MPI_Recv(AS) -> Check(Err) --------------------------- There is a race between when the error is returned to Thread A, and when Thread B posts the blocking MPI_Recv(AS). In the model proposed by UTK, the MPI_Recv(AS) in Thread B would be in a separate epoch since the error was returned to the user already. So this receive would block, even if the user did not want it to. The user would have to lock around the MPI_Recv(AS) to allow only one thread to call it at a time to avoid this race condition. In the RTS model, the MPI_Recv(AS) in Thread B would return an error until the user reenables the ANY_SOURCE receives. So the user does not need to lock around the MPI_Recv(), but only around the reenable_anysource (per the example in the RTS proposal). Maybe this would be a good example to include as support for a rationale? -- Josh > > > On Dec 5, 2011, at 3:20 PM, Graham, Richard L. wrote: > >>>> 17.6: MPI_ANY_SOURCE >>>> ------------------------- >>>> Problem: The model is too complex and can be simplified. >>>> >>>> A blocking MPI_Recv(ANY) will return an error when a new failure >>>> occurs. A nonblocking MPI_Irecv(ANY) request, during the >>>> MPI_Wait/Test, will return a 'warning' (MPI_ERR_ANY_SOURCE_DISABLED) >>>> and not complete when a new failure occurs. >>>> >>>> There was a debate if the user needs to call another function (like >>>> MPI_Comm_reenable_any_source) to reactivate the ANY_SOURCE receives, >>>> or if the fact that the error was returned to the user is sufficient >>>> to reactivate ANY_SOURCE. The ones that returned a 'warning' error can >>>> be matched/completed if while the user is handling the error a >>>> matching send arrives. So this kind of just works like an interrupt. >>>> >>>> Additionally, once an error is returned should we remove the >>>> protection that "Any new MPI_ANY_SOURCE receive operations using a >>>> communicator with MPI_ANY_SOURCE receive operations disabled will not >>>> be posted, and marked with an error code of the class >>>> MPI_ERR_PROC_FAIL_STOP." So new MPI_Recv(ANY) would post successfully. >>>> >>>> The sentiment is that the user is notified of the process failure via >>>> the return code from the MPI_Wait/Test, and if they do another >>>> MPI_Wait/Test or post a new ANY_SOURCE receive they implicitly >>>> acknowledge the failed process. To avoid a race condition where >>>> another process fails between the error return and the MPI_Wait/Test, >>>> the library needs to associate an 'epoch' value with the communicator >>>> to make sure that the next MPI_Wait/Test returns another error. >>>> >>>> I like the idea of the user doing an explicit second operation, like >>>> MPI_Comm_reenable_any_source, since then we know that the user is >>>> aware of the failed group (and seems more MPI-like to me). However, I >>>> like the more flexible semantics. So I'm a bit split on this one. >>>> >>> >>> There's still a race with threads: ï¿½A thread posts a blocking AS receive. ï¿½The receive completes with an error because some process in the comm failed. ï¿½Another thread posts a blocking AS receive immediately after the receive on the first thread returns. >>> >>> Even though, from the MPI library's point of view, the application has been "informed" of the failure when the first receive returned with an error, there wasn't enough time for the application to react to the error and prevent the second thread from posting the receive. >> >> [rich] I expect the user would get an error in both cases. ï¿½Is there a problem with this ? ï¿½As long as the user does not get a response that would have it get into a deadlock state, I think we are OK. ï¿½If we say that we are in an error state and are not, this is OK, the user can "fix" this. ï¿½If we enter into into the call and think we are not in an error state, but really are, I think we are also OK, if we keep track of epochs. >> >> Rich > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ Noon EDT
by Josh Hursey 06 Dec '11

06 Dec '11

Reminder that we have a teleconf tomorrow, Wednesday 12/7/2011, at Noon EDT. Date: December 7, 2011 Time: Noon EST/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Discussion of notes from UTK meeting http://lists.mpi-forum.org/mpi3-ft/2011/12/0922.php - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main - Other items? Additional Agenda Items [as time permits]: - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Piggybacking? (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Dec. 14, 2011 (Teleconf) * Jan. 4, 2012 (Teleconf) * Jan. 9, 2012 (MPI Forum Meeting) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] A few notes from UTK about the RTS proposal
by Darius Buntinas 05 Dec '11

05 Dec '11

Re-reading what I wrote, I think I wasn't clear. I was referring to the suggestion that reenable_anysource was not needed because the user was already notified of the failure when the first IRecv(ANYSOURCE) returned an error. I don't think that's sufficient when dealing with threads (see my example). If I misunderstood what was suggested at the meeting, then ignore what I said. I do agree with Rich that both should return an error. And any subsequent anysources should as well until reenable_anysource (or whatever we name it) gets called. Is there another way to deal with the deadlock? We can, of course, punt and leave it up to the user as I suggested later. -d On Dec 5, 2011, at 3:20 PM, Graham, Richard L. wrote: >>> 17.6: MPI_ANY_SOURCE >>> ------------------------- >>> Problem: The model is too complex and can be simplified. >>> >>> A blocking MPI_Recv(ANY) will return an error when a new failure >>> occurs. A nonblocking MPI_Irecv(ANY) request, during the >>> MPI_Wait/Test, will return a 'warning' (MPI_ERR_ANY_SOURCE_DISABLED) >>> and not complete when a new failure occurs. >>> >>> There was a debate if the user needs to call another function (like >>> MPI_Comm_reenable_any_source) to reactivate the ANY_SOURCE receives, >>> or if the fact that the error was returned to the user is sufficient >>> to reactivate ANY_SOURCE. The ones that returned a 'warning' error can >>> be matched/completed if while the user is handling the error a >>> matching send arrives. So this kind of just works like an interrupt. >>> >>> Additionally, once an error is returned should we remove the >>> protection that "Any new MPI_ANY_SOURCE receive operations using a >>> communicator with MPI_ANY_SOURCE receive operations disabled will not >>> be posted, and marked with an error code of the class >>> MPI_ERR_PROC_FAIL_STOP." So new MPI_Recv(ANY) would post successfully. >>> >>> The sentiment is that the user is notified of the process failure via >>> the return code from the MPI_Wait/Test, and if they do another >>> MPI_Wait/Test or post a new ANY_SOURCE receive they implicitly >>> acknowledge the failed process. To avoid a race condition where >>> another process fails between the error return and the MPI_Wait/Test, >>> the library needs to associate an 'epoch' value with the communicator >>> to make sure that the next MPI_Wait/Test returns another error. >>> >>> I like the idea of the user doing an explicit second operation, like >>> MPI_Comm_reenable_any_source, since then we know that the user is >>> aware of the failed group (and seems more MPI-like to me). However, I >>> like the more flexible semantics. So I'm a bit split on this one. >>> >> >> There's still a race with threads: A thread posts a blocking AS receive. The receive completes with an error because some process in the comm failed. Another thread posts a blocking AS receive immediately after the receive on the first thread returns. >> >> Even though, from the MPI library's point of view, the application has been "informed" of the failure when the first receive returned with an error, there wasn't enough time for the application to react to the error and prevent the second thread from posting the receive. > > [rich] I expect the user would get an error in both cases. Is there a problem with this ? As long as the user does not get a response that would have it get into a deadlock state, I think we are OK. If we say that we are in an error state and are not, this is OK, the user can "fix" this. If we enter into into the call and think we are not in an error state, but really are, I think we are also OK, if we keep track of epochs. > > Rich

1 0