September 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Updates to ft chapter
by Josh Hursey 15 Sep '11

15 Sep '11

I made some minor-ish edits. I attached the diff to this email for review. Feel free to commit it if you think it is good to go. Some larger items that I did not want to change/adjust before discussing with the group: 17.1: Does the Advice to Implementors buy us anything? Should it be reworded? 17.2: Do we need the definitions of 'error' and 'failure'? We don't rely on these definitions in the text beyond their previously implied definitions in the standard. If we do not need them, then it might be good to drop them to reduce complexity. 17.5: It was suggested that we try to clarify this paragraph with the Rationale. Any suggestions? 17.5.2: Should we go ahead and pull the Advice to implementors regarding the return value of mpiexec into a separate ticket? Or should we keep it in the document and pull it if we get pushback? (I think on the call we decided the latter, but I forget now). 17.7: In the second Rationale paragraph. I moved the first sentence to 17.7.1. But I think we can drop the rest of the rationale. I do not know if it is terribly instructive. 17.7.1: I updated the rational to account for the MPI_Reduce numerical stability recommendation. --------------- Rationale. The MPI_COMM_VALIDATE and MPI_ICOMM_VALIDATE operations provide the MPI implementation an opportunity to restructure collective communication patterns before the communicator is used by the alive process. This may allow for improved collective performance after process failure. It should be noted such optimizations might change the consistency recommendation for MPI_REDUCE in the advice to implementors in Section ??. It is strongly recommended that the consistency recommendation hold for MPI_REDUCE between consecutive collective activations of a communicator using a collective validation operation (e.g, MPI_COMM_VALIDATE). (End of rationale.) --------------- 17.8.2: Note that I moved the Advice to users regarding libraries to here, per the teleconf. 17.12.1: Added back the Advice to users regarding the 'sync-barrier-sync' semantic for MPI_File_validate. Thanks, Josh On Wed, Sep 14, 2011 at 1:58 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > I've made some changes we discussed on the phone this morning. You can find the latest pdf here (or at the bottom of the "Modified run-through stabilization" page on the wiki): > https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… > > Here's a summary of the changes: > * Changed MPI_ERR_RANK_FAIL_STOP to MPI_ERR_PROC_FAIL_STOP (because a "rank" doesn't fail, a "processes" does) > * Fixed up usage of rank vs process in the chapter. > * Removed MPI_COMM_COLLECTIVES_ENABLED function because it returns local version of a global state which is meaningless for applications. > * Added MPI_COMM_ANY_SOURCE_ENABLED > * Moved the definition of MPI_COMM_VALIDATE et.al. earlier in the section, and added a new subsection. > > Please look over my changes, especially how I rearranged the collectives section for the definition of MPI_COMM_VALIDATE, and let me know if they look OK. > > Thanks! > -d > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Sur, Sayantan 15 Sep '11

15 Sep '11

> -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey > Sent: Thursday, September 15, 2011 6:13 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] Multiple Communicator Version of > MPI_Comm_validate > > On Wed, Sep 14, 2011 at 4:17 PM, Sur, Sayantan <sayantan.sur(a)intel.com> > wrote: > > Hi Josh, > > > >> > >> Workaround: > >> -------------------- > >> Call MPI_Comm_validate over all of the communicators individually. > >> This would involve 'num_comms' collective operations and likely > impede > >> scalability. > >> > >> for(i=0; i < num_comms; ++i) { > >> ï¿½ MPI_Comm_validate(comm[i], failed_grps[i]); > >> } > >> > > > > Would it not be possible for the app to create a communicator that is > the union of all the processes, and subsequently call validate only on > that 'super' communicator? I hope I am not missing something from your > example. > > In the current spec, no. MPI_Comm_validate only changes the state of > the communicator passed. We probably want to create a new API like > MPI_Comm_validate_many() to host these new semantics. I agree with your comments. I agree that calling 'validate' on communicators somehow not owned (through heritage, etc.) is tricky business as it changes the semantics. I guess my question is that in your example comm[0..num_comms-1] are 'owned' by the layer of application code that is calling validate? There are two cases here: i) how to optimize validation for a set of possibly overlapping communicators (all owned by one layer of application code [library]) ii) how to optimize validation for a set of communicators (possibly heritage linked by dup) across layers of libraries ... thus eliminating the requirement of each library to validate its own communicator Which of these cases does MPI_Comm_validate_many() aim to optimize? Thanks. > > It is important to remember that the validate operation changes the > communicator (primarily just the 'are_collectives_enabled' flag on the > communicator), and not anything to do with elements of the group that > form it. > > Currently after creation, a communicator does not need to track from > which communicators it was created (at least that's the way I > understand it). So creating a super communicator and calling > MPI_Comm_validate_many() on that would require such tracking to have > the validation propagate to all of the communicators that built it. So > we could do it, but the additional state tracking would force > additional memory consumption even if the operation is never used, > which is slightly problematic. > > > > > > I liked your Option B as such, however, as you point out, it has > significant problems in case of applications consisting of several > layers of libraries. > > > Thinking through your question above, I think Option B would require > that we track the heritage of communicators after creation, which > would increase memory consumption. It would also require us to > maintain that linkage across communicator destruction. For example, > ----------- > MPI_Comm_dup(MPI_COMM_WORLD, commA); > // MCW is linked to commA > MPI_Comm_dup(commA, commB); > // MCW is linked to commA > // commA is linked to commB > MPI_Comm_dup(commB, commC); > // MCW is linked to commA > // commA is linked to commB > // commB is linked to commC > MPI_Comm_free(commB); > // MCW is linked to commA > // commA is linked to commC (since commB is now gone) > ------------ > > In the discussion so far it seems that the inheritance is only one > way. Meaning that in the example above calling > MPI_Comm_validate_many() on commA would validate commC (and commB if > it is still around), but not MPI_COMM_WORLD. Is that what we are > looking for, or do we want it to be more complete? > > > The explicit linking in Option C puts the user in more control over > the overhead of tracking connections between communicators, but has > other issues. :/ > > -- Josh > > > > > > Sayantan. > > > >> > >> Option A: > >> Array of communicators > >> -------------------- > >> MPI_Comm_validate_many(comm[], num_comms, failed_grp) > >> Validate 'num_comms' communicators, and return a failed group. > >> ï¿½ - or - > >> MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) > >> Validate 'num_comms' communicators, and return a failed group for > each > >> communicator. > >> ---- > >> > >> In this version of the operation the user passes in an array of > >> pointers to communicators. Since communicators are not often created > >> in a contiguous array, pointers to communications should probably be > >> used. The failed_grps is an array of failures in each of those > >> communicators. > >> > >> Some questions: > >> ï¿½* Should all processes pass in the same set of communicators at all > >> processes? > >> ï¿½* Should all communicators be duplicates or subsets of one another? > >> ï¿½* Does this operation run the risk of a circular dependency if the > >> user does not pass in the same set of communicators at all > >> participating processes? Is that something the MPI library should > >> protect the application from? > >> > >> > >> Option B: > >> Implicit inherited validation > >> -------------------- > >> MPI_Comm_validate_many(comm, failed_grp) > >> ---- > >> > >> The idea is to add an additional semantic (or maybe new API) to > allows > >> the validation of a communicator to automatically validates all > >> communicators created from it (only dups and subsets of it?). > >> > >> The problem with this is that if an application calls > >> MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of > >> communicators that libraries might be using internally without > >> notification in those libraries. So this breaks the abstraction > >> barrier between the two in possibly a dangerous way. > >> > >> Some questions: > >> ï¿½* Are there some other semantics that we can add to help protect > >> libraries? (e.g., after implicit validation the first use of the > >> communicator will return a special error code indicating that the > >> communicator has been adjusted). > >> ï¿½* Are there thread safety issues involved with this? (e.g., the > >> library operates in a concurrent thread with its own duplicate of > the > >> communicator. The application does not know about or control the > >> concurrent thread but calls MPI_Comm_validate on its own > communicator > >> and implicitly changes the semantics of the duplicate communicator.) > >> ï¿½* It is only through the call to MPI_Comm_validate that we can > >> provide a uniform group of failed processes globally known. For > those > >> that were implicitly validated, do we need to provide a way to > access > >> this group after the call? Does this have implications on the amount > >> of storage required for this semantic? > >> > >> > >> Option C: > >> Explicit inherited validation > >> -------------------- > >> MPI_Comm_validate_link(commA, commB); > >> MPI_Comm_validate_many(commA, failed_grp) > >> /* Implies MPI_Comm_validate(commB, NULL) */ > >> > >> MPI_Comm_validate(commA, failed_grp) > >> /* Does not imply MPI_Comm_validate(commB, NULL) */ > >> ---- > >> > >> In this version the application explicitly links communicators. This > >> prevents an application from implicitly altering derived > communicators > >> out of their scope (e.g., in use by other libraries). > >> > >> Some questions: > >> ï¿½* It is only through the call to MPI_Comm_validate that we can > >> provide a uniform group of failed processes globally known. For > those > >> that were implicitly validated, do we need to provide a way to > access > >> this group after the call (e.g., for commB)? Does this have > >> implications on the amount of storage required for this semantic? > >> ï¿½* Do we need a mechanism to 'unlink' communicators? Or determine > >> which communicators are linked? > >> ï¿½* Can a communicator be linked to multiple other communicators? > >> ï¿½* Is the linking a unidirectional operation? (so in the example > above > >> validating commB does not validate commA unless there is a separate > >> MPI_Comm_validate_link(commB, commA) call) > >> > >> > >> Option D: > >> Other > >> -------------------- > >> Something else... > >> > >> > >> Thoughts? > >> > >> -- Josh > >> > >> > >> -- > >> Joshua Hursey > >> Postdoctoral Research Associate > >> Oak Ridge National Laboratory > >> http://users.nccs.gov/~jjhursey > >> _______________________________________________ > >> mpi3-ft mailing list > >> mpi3-ft(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Josh Hursey 15 Sep '11

15 Sep '11

On Wed, Sep 14, 2011 at 4:17 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: > Hi Josh, > >> >> Workaround: >> -------------------- >> Call MPI_Comm_validate over all of the communicators individually. >> This would involve 'num_comms' collective operations and likely impede >> scalability. >> >> for(i=0; i < num_comms; ++i) { >> ï¿½ MPI_Comm_validate(comm[i], failed_grps[i]); >> } >> > > Would it not be possible for the app to create a communicator that is the union of all the processes, and subsequently call validate only on that 'super' communicator? I hope I am not missing something from your example. In the current spec, no. MPI_Comm_validate only changes the state of the communicator passed. We probably want to create a new API like MPI_Comm_validate_many() to host these new semantics. It is important to remember that the validate operation changes the communicator (primarily just the 'are_collectives_enabled' flag on the communicator), and not anything to do with elements of the group that form it. Currently after creation, a communicator does not need to track from which communicators it was created (at least that's the way I understand it). So creating a super communicator and calling MPI_Comm_validate_many() on that would require such tracking to have the validation propagate to all of the communicators that built it. So we could do it, but the additional state tracking would force additional memory consumption even if the operation is never used, which is slightly problematic. > > I liked your Option B as such, however, as you point out, it has significant problems in case of applications consisting of several layers of libraries. Thinking through your question above, I think Option B would require that we track the heritage of communicators after creation, which would increase memory consumption. It would also require us to maintain that linkage across communicator destruction. For example, ----------- MPI_Comm_dup(MPI_COMM_WORLD, commA); // MCW is linked to commA MPI_Comm_dup(commA, commB); // MCW is linked to commA // commA is linked to commB MPI_Comm_dup(commB, commC); // MCW is linked to commA // commA is linked to commB // commB is linked to commC MPI_Comm_free(commB); // MCW is linked to commA // commA is linked to commC (since commB is now gone) ------------ In the discussion so far it seems that the inheritance is only one way. Meaning that in the example above calling MPI_Comm_validate_many() on commA would validate commC (and commB if it is still around), but not MPI_COMM_WORLD. Is that what we are looking for, or do we want it to be more complete? The explicit linking in Option C puts the user in more control over the overhead of tracking connections between communicators, but has other issues. :/ -- Josh > > Sayantan. > >> >> Option A: >> Array of communicators >> -------------------- >> MPI_Comm_validate_many(comm[], num_comms, failed_grp) >> Validate 'num_comms' communicators, and return a failed group. >> ï¿½ - or - >> MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) >> Validate 'num_comms' communicators, and return a failed group for each >> communicator. >> ---- >> >> In this version of the operation the user passes in an array of >> pointers to communicators. Since communicators are not often created >> in a contiguous array, pointers to communications should probably be >> used. The failed_grps is an array of failures in each of those >> communicators. >> >> Some questions: >> ï¿½* Should all processes pass in the same set of communicators at all >> processes? >> ï¿½* Should all communicators be duplicates or subsets of one another? >> ï¿½* Does this operation run the risk of a circular dependency if the >> user does not pass in the same set of communicators at all >> participating processes? Is that something the MPI library should >> protect the application from? >> >> >> Option B: >> Implicit inherited validation >> -------------------- >> MPI_Comm_validate_many(comm, failed_grp) >> ---- >> >> The idea is to add an additional semantic (or maybe new API) to allows >> the validation of a communicator to automatically validates all >> communicators created from it (only dups and subsets of it?). >> >> The problem with this is that if an application calls >> MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of >> communicators that libraries might be using internally without >> notification in those libraries. So this breaks the abstraction >> barrier between the two in possibly a dangerous way. >> >> Some questions: >> ï¿½* Are there some other semantics that we can add to help protect >> libraries? (e.g., after implicit validation the first use of the >> communicator will return a special error code indicating that the >> communicator has been adjusted). >> ï¿½* Are there thread safety issues involved with this? (e.g., the >> library operates in a concurrent thread with its own duplicate of the >> communicator. The application does not know about or control the >> concurrent thread but calls MPI_Comm_validate on its own communicator >> and implicitly changes the semantics of the duplicate communicator.) >> ï¿½* It is only through the call to MPI_Comm_validate that we can >> provide a uniform group of failed processes globally known. For those >> that were implicitly validated, do we need to provide a way to access >> this group after the call? Does this have implications on the amount >> of storage required for this semantic? >> >> >> Option C: >> Explicit inherited validation >> -------------------- >> MPI_Comm_validate_link(commA, commB); >> MPI_Comm_validate_many(commA, failed_grp) >> /* Implies MPI_Comm_validate(commB, NULL) */ >> >> MPI_Comm_validate(commA, failed_grp) >> /* Does not imply MPI_Comm_validate(commB, NULL) */ >> ---- >> >> In this version the application explicitly links communicators. This >> prevents an application from implicitly altering derived communicators >> out of their scope (e.g., in use by other libraries). >> >> Some questions: >> ï¿½* It is only through the call to MPI_Comm_validate that we can >> provide a uniform group of failed processes globally known. For those >> that were implicitly validated, do we need to provide a way to access >> this group after the call (e.g., for commB)? Does this have >> implications on the amount of storage required for this semantic? >> ï¿½* Do we need a mechanism to 'unlink' communicators? Or determine >> which communicators are linked? >> ï¿½* Can a communicator be linked to multiple other communicators? >> ï¿½* Is the linking a unidirectional operation? (so in the example above >> validating commB does not validate commA unless there is a separate >> MPI_Comm_validate_link(commB, commA) call) >> >> >> Option D: >> Other >> -------------------- >> Something else... >> >> >> Thoughts? >> >> -- Josh >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Sur, Sayantan 14 Sep '11

14 Sep '11

Hi Josh, > > Workaround: > -------------------- > Call MPI_Comm_validate over all of the communicators individually. > This would involve 'num_comms' collective operations and likely impede > scalability. > > for(i=0; i < num_comms; ++i) { > MPI_Comm_validate(comm[i], failed_grps[i]); > } > Would it not be possible for the app to create a communicator that is the union of all the processes, and subsequently call validate only on that 'super' communicator? I hope I am not missing something from your example. I liked your Option B as such, however, as you point out, it has significant problems in case of applications consisting of several layers of libraries. Sayantan. > > Option A: > Array of communicators > -------------------- > MPI_Comm_validate_many(comm[], num_comms, failed_grp) > Validate 'num_comms' communicators, and return a failed group. > - or - > MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) > Validate 'num_comms' communicators, and return a failed group for each > communicator. > ---- > > In this version of the operation the user passes in an array of > pointers to communicators. Since communicators are not often created > in a contiguous array, pointers to communications should probably be > used. The failed_grps is an array of failures in each of those > communicators. > > Some questions: > * Should all processes pass in the same set of communicators at all > processes? > * Should all communicators be duplicates or subsets of one another? > * Does this operation run the risk of a circular dependency if the > user does not pass in the same set of communicators at all > participating processes? Is that something the MPI library should > protect the application from? > > > Option B: > Implicit inherited validation > -------------------- > MPI_Comm_validate_many(comm, failed_grp) > ---- > > The idea is to add an additional semantic (or maybe new API) to allows > the validation of a communicator to automatically validates all > communicators created from it (only dups and subsets of it?). > > The problem with this is that if an application calls > MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of > communicators that libraries might be using internally without > notification in those libraries. So this breaks the abstraction > barrier between the two in possibly a dangerous way. > > Some questions: > * Are there some other semantics that we can add to help protect > libraries? (e.g., after implicit validation the first use of the > communicator will return a special error code indicating that the > communicator has been adjusted). > * Are there thread safety issues involved with this? (e.g., the > library operates in a concurrent thread with its own duplicate of the > communicator. The application does not know about or control the > concurrent thread but calls MPI_Comm_validate on its own communicator > and implicitly changes the semantics of the duplicate communicator.) > * It is only through the call to MPI_Comm_validate that we can > provide a uniform group of failed processes globally known. For those > that were implicitly validated, do we need to provide a way to access > this group after the call? Does this have implications on the amount > of storage required for this semantic? > > > Option C: > Explicit inherited validation > -------------------- > MPI_Comm_validate_link(commA, commB); > MPI_Comm_validate_many(commA, failed_grp) > /* Implies MPI_Comm_validate(commB, NULL) */ > > MPI_Comm_validate(commA, failed_grp) > /* Does not imply MPI_Comm_validate(commB, NULL) */ > ---- > > In this version the application explicitly links communicators. This > prevents an application from implicitly altering derived communicators > out of their scope (e.g., in use by other libraries). > > Some questions: > * It is only through the call to MPI_Comm_validate that we can > provide a uniform group of failed processes globally known. For those > that were implicitly validated, do we need to provide a way to access > this group after the call (e.g., for commB)? Does this have > implications on the amount of storage required for this semantic? > * Do we need a mechanism to 'unlink' communicators? Or determine > which communicators are linked? > * Can a communicator be linked to multiple other communicators? > * Is the linking a unidirectional operation? (so in the example above > validating commB does not validate commA unless there is a separate > MPI_Comm_validate_link(commB, commA) call) > > > Option D: > Other > -------------------- > Something else... > > > Thoughts? > > -- Josh > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] fault-tolerant collectives
by Josh Hursey 14 Sep '11

14 Sep '11

Note: I started a new thread on the multiple MPI_Comm_validate operation, since it is a divergence in this thread. http://lists.mpi-forum.org/mpi3-ft/2011/09/0844.php For fault tolerant collectives: So how about a function signature like: MPI_*_ft(..., fail_grp) So postfix the function with '_ft' to differentiate them from the traditional collectives. Semantically the fault tolerant collectives will (to the greatest possible extent) work around existing and emerging process failures to complete the operation successfully. Upon successful completion, all processes are returned a consistent version of 'fail_grp' containing those failed processes that did not contribute to the completion of the collective. If the collective is not successful, then all processes will be returned some error code. So function signatures like: ------------------- MPI_Barrier_ft(..., fail_grp); MPI_Bcast_ft(..., fail_grp); MPI_Gather{v}_ft(..., fail_grp); MPI_Scatter{v}_ft(..., fail_grp); MPI_Allgather{v}_ft(..., fail_grp); MPI_Alltoall{v|w}_ft(..., fail_grp); MPI_Reduce_ft(..., fail_grp); MPI_Allreduce_ft(..., fail_grp); MPI_Reduce_scatter_ft(..., fail_grp); MPI_Reduce_scatter_block_ft(..., fail_grp); MPI_Scan_ft(..., fail_grp); MPI_Exscan_ft(..., fail_grp); + nonblocking versions of each. ------------------- Note that MPI_Comm_validate() and MPI_Barrier_ft() would algorithmically behave exactly the same, except that the former would 're-enable traditional collectives'. Some questions: * Is MPI_Barrier_ft() useful to provide? * Should we require that the communicator be collectively active before calling these collectives (like we do for traditional collectives)? Since they can work around process failure, we might want to consider lifting this restriction. * Do we want to introduce an object like MPI_GROUP_IGNORE (similar to MPI_STATUS_IGNORE) that allows a user to ignore the fail_grp parameter? For example a user calling allreduce may not care which processes failed as long as a result was reached uniformly. * Are there any constraints on the output buffers for these collectives? For example, a reduce that must work around failures might need some clarification about the numerical stability of the output. And MPI_Op functions might be called multiple times for the same reduction depending if the collective needs to be re-executed to provide the necessary semantics. * The traditional collectives do not return uniformly, and are only required to be fault aware. These new collectives must return uniformly, and be fault tolerant. Do we want the ability to have traditional collectives return uniformly, but only be required to be fault aware? Something like a half-step between traditional fault-aware collectives and fully fault tolerant collectives. We allow for this in an 'advice to implementors', but do we need it to be an explicit capability? Thoughts? -- Josh On Mon, Sep 12, 2011 at 1:40 PM, Bronis R. de Supinski <bronis(a)llnl.gov> wrote: > > It would be useful from a scalability standpoint. > I have not thought through how much trouble it > might cause from a programmability standpoint. > It just seems the inherited mechanism is useful > in costs. It might be useful to have an attribute > that says to return an error (or some other notification) > if it has been validated through inheritance. Then > the library fixes could potentially just be local > instead of repeating the interprocess work... > > On Mon, 12 Sep 2011, Darius Buntinas wrote: > >> Are you're saying that the app might want to validate a library's >> (internal) communicator for it? >> >> In general, how safe would it be to do that if the library isn't expecting >> it? ï¿½I suppose we could do something like an inherited attribute that makes >> all subcommunicators automatically validate when it's parent does. >> >> -d >> >> On Sep 12, 2011, at 12:12 PM, Bronis R. de Supinski wrote: >> >>> >>> Suppose some communicators are used in libraries >>> (duplication of MPI_COMM_WORLD is very common). >>> How would any part of the code know about all of >>> them? Sure, you could use the profiling interface >>> to intercept calls to track them but usual application >>> code would not have any record of the handles... >>> >>> On Mon, 12 Sep 2011, Darius Buntinas wrote: >>> >>>> >>>> On Sep 11, 2011, at 4:45 PM, Josh Hursey wrote: >>>> >>>>> On Fri, Sep 9, 2011 at 7:57 PM, Graham, Richard L. <rlgraham(a)ornl.gov> >>>>> wrote: >>>>>> >>>>>> I have been talking a reasonable amount with apps folks lately about >>>>>> this proposal, and they first response is often one of shock, as it is not >>>>>> quite what folks initially expect. ï¿½However, once one explains the >>>>>> background for the proposal, people tend to accept the notions. >>>>> >>>>> Can you explain a bit more about what they were shocked by? Was it the >>>>> general notion of application involved FT, or the interface not being >>>>> what they expected/needed? >>>>> >>>>> >>>>>> I agree that we need to define a mechanism for specifying return codes >>>>>> - uniform among surviving ranks, or locally determined types. ï¿½However, I do >>>>>> believe that we need to add the second set of collectives into 3.0. ï¿½We have >>>>>> mentioned this as an option for several years (actually since the inception >>>>>> of the group almost 4 years ago), but as a working group never did something >>>>>> explicit about this. ï¿½There is a reasonable number of apps folks that expect >>>>>> this type of collective communications. >>>>> >>>>> >>>>> It shouldn't be to difficult to specify/add, and a prototype >>>>> implementation would be trivial though maybe inefficient at first. Is >>>>> this something that we should put in the Stabilization proposal or >>>>> bring in as a separate ticket directly afterward? >>>>> >>>>> Separating the two keeps the initial proposal simpler and users can >>>>> get this functionality by wrapping existing collectives in >>>>> comm_validate calls (not efficient, but functional). Keeping them >>>>> together allows us to address a know interface optimization that >>>>> applications want in the first pass. >>>> >>>> It seems that this can be done in the app itself (or a stand-alone >>>> utility library): >>>> >>>> int MYMPI_Bcast_ft(...) { >>>> ï¿½while (1) { >>>> ï¿½ ï¿½ ï¿½ret = MPI_Bcast(...); >>>> ï¿½ ï¿½ ï¿½if (ret == MPI_ERR_RANK_FAILSTOP) { >>>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ret == MPI_Comm_validate(...); >>>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (ret) return ret; >>>> ï¿½ ï¿½ ï¿½} else { >>>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½return ret; >>>> ï¿½ ï¿½ ï¿½} >>>> ï¿½} >>>> } >>>> >>>> ...or something like that. ï¿½If an implementation implements it's >>>> collectives in a FT manner, then the validate call can be close to a noop >>>> (maybe a simple allreduce to make sure the failed_groups are the same). >>>> >>>>>> One other thing that came up yesterday (I have given 2 talks about the >>>>>> FT stuff in Kobe this week) is that it would be good to be able to specify >>>>>> multiple communicators to mpi_comm_validate(), especially, since a common >>>>>> motif is to dup an existing communicator to isolate communication. ï¿½This is >>>>>> really the only way that I can think of to avoid un-needed global >>>>>> communication, if more than one communicator is of interest to the app. >>>>> >>>>> >>>>> I've had a couple applications ask about this as well. >>>>> >>>>> The group has talked about such an interface a few times now, and keep >>>>> getting stuck on specifying the interface and semantics of such an >>>>> operation. Did they want a function that would take an array of >>>>> communicators to validate, or have the validation of one communicator >>>>> be inherited by all of the derived communicators? >>>>> >>>>> I think the array of communicators interface seems like the easiest to >>>>> use, and makes it easier to protect libraries. But that leads us to >>>>> the question, do all processes (union of processes from all >>>>> communicators specified?) have to supply the same set of >>>>> communicators? If not, do we run the risk of a circular dependency >>>>> causing the call to deadlock? We might be able to pass the >>>>> responsibility to avoid such problems off to the user. >>>> >>>> Yeah, I think it would be easiest to require that the same communicators >>>> are specified. ï¿½And I don't think that would be too difficult for the users >>>> to handle. >>>> >>>> When we talked about "linking" communicators with attributes or >>>> something, such that when one communicator is validated, all of the linked >>>> communicators are also validated. ï¿½In that case we said that the all of the >>>> linked communicators must be subsets (or dups) of the communicator they're >>>> linked to. >>>> >>>> We could have a similar "subset" restriction for the >>>> array-of-communicators-to-comm_validate option. >>>> >>>>> I'm game for trying to specify this again. The group decided to push >>>>> this off to a follow on ticket because it can be achieved (though >>>>> inefficiently) by making a call to comm_validate for each of the >>>>> communicators, and we had trouble specifying it correctly. So the >>>>> question again is should be keep it as a separate ticket or add it to >>>>> the stabilization proposal? >>>> >>>> I'm kinda leaning toward adding it to the current proposal, but I don't >>>> want to delay the proposal. >>>> -d >>>> >>>> >>>>> Thoughts? >>>>> >>>>> -- Josh >>>>> >>>>> >>>>>> >>>>>> Rich >>>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org >>>>>> [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>>> Sent: Friday, September 09, 2011 4:39 PM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>> Subject: Re: [Mpi3-ft] fault-tolerant collectives >>>>>> >>>>>> >>>>>> OK, that makes sense. ï¿½I'll fix up that text. >>>>>> >>>>>> Thanks, >>>>>> -d >>>>>> >>>>>> On Sep 9, 2011, at 3:36 PM, Josh Hursey wrote: >>>>>> >>>>>>> I think that we want to say that an implementation may provide >>>>>>> uniform >>>>>>> return codes from collectives, but are not required to do so. So this >>>>>>> makes then fault tolerant-ish - in the sense that they have to work >>>>>>> around failure to return error codes consistently, but not that they >>>>>>> finish the collective successfully even if new process failures >>>>>>> emerge >>>>>>> during the collectives (that would undermine the semantic protections >>>>>>> we are putting in place). >>>>>>> >>>>>>> We should probably not say 'fault tolerant collectives' in the >>>>>>> current >>>>>>> proposal so we don't confuse things. Maybe 'collectives that provide >>>>>>> uniform return codes'? >>>>>>> >>>>>>> >>>>>>> If we want truly fault tolerant collectives (like those described >>>>>>> below), then I think we should introduce a different set of >>>>>>> functions. >>>>>>> The functions should probably return a group of processes that either >>>>>>> did or did not participate in creating the final result. Something >>>>>>> like: >>>>>>> MPI_Reduce_ft(..., &group); >>>>>>> >>>>>>> I think the true fault tolerant collectives should be left to a >>>>>>> follow >>>>>>> on ticket since there is a need, but can be easily added as a second >>>>>>> step. >>>>>>> >>>>>>> -- Josh >>>>>>> >>>>>>> On Fri, Sep 9, 2011 at 4:17 PM, Darius Buntinas >>>>>>> <buntinas(a)mcs.anl.gov> wrote: >>>>>>>> >>>>>>>> We discussed the option of allowing an implementation to provide >>>>>>>> fault tolerant (not just fault-aware) collectives. ï¿½The idea is that even >>>>>>>> when a process fails, collectives will continue to operate correctly (modulo >>>>>>>> the failed process). >>>>>>>> >>>>>>>> Does this imply that the communicator will never become collectively >>>>>>>> inactive? >>>>>>>> >>>>>>>> If no, then what's the point of ft collectives? >>>>>>>> >>>>>>>> If yes, then the application may never get notification that a >>>>>>>> process has failed and collectives are now running one short. ï¿½Is this what >>>>>>>> we really want? >>>>>>>> >>>>>>>> -d >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Joshua Hursey >>>>>>> Postdoctoral Research Associate >>>>>>> Oak Ridge National Laboratory >>>>>>> hxxp://users.nccs.gov/~jjhursey >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Joshua Hursey >>>>> Postdoctoral Research Associate >>>>> Oak Ridge National Laboratory >>>>> http://users.nccs.gov/~jjhursey >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Updates to ft chapter
by Darius Buntinas 14 Sep '11

14 Sep '11

I've made some changes we discussed on the phone this morning. You can find the latest pdf here (or at the bottom of the "Modified run-through stabilization" page on the wiki): https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… Here's a summary of the changes: * Changed MPI_ERR_RANK_FAIL_STOP to MPI_ERR_PROC_FAIL_STOP (because a "rank" doesn't fail, a "processes" does) * Fixed up usage of rank vs process in the chapter. * Removed MPI_COMM_COLLECTIVES_ENABLED function because it returns local version of a global state which is meaningless for applications. * Added MPI_COMM_ANY_SOURCE_ENABLED * Moved the definition of MPI_COMM_VALIDATE et.al. earlier in the section, and added a new subsection. Please look over my changes, especially how I rearranged the collectives section for the definition of MPI_COMM_VALIDATE, and let me know if they look OK. Thanks! -d

1 0

[Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Josh Hursey 14 Sep '11

14 Sep '11

(I decided to split this into its own thread of discussion from what was developing on [1]) Recently it was mentioned on the list [1] that users would like the ability to validate multiple communicators (or windows or file handles) in a single collective call. We have discussed this a few times in the past. Defining the interface and semantics appropriately has prevented us from defining such an operation so far. So though we saw the desire for such an operation we pushed it to a separate follow-on ticket until we could get it right. [1] http://lists.mpi-forum.org/mpi3-ft/2011/09/0836.php Problem: -------------------- MPI_Comm_validate(comm, failed_grp) The current operation validates one communicator, and returns the group of globally agreed upon failed processes associated with that communicator. Users would like the ability to, in a single collective call, validate multiple communicators. This would likely result in better performance over calling MPI_Comm_validate over each individual communicator. Workaround: -------------------- Call MPI_Comm_validate over all of the communicators individually. This would involve 'num_comms' collective operations and likely impede scalability. for(i=0; i < num_comms; ++i) { MPI_Comm_validate(comm[i], failed_grps[i]); } Option A: Array of communicators -------------------- MPI_Comm_validate_many(comm[], num_comms, failed_grp) Validate 'num_comms' communicators, and return a failed group. - or - MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) Validate 'num_comms' communicators, and return a failed group for each communicator. ---- In this version of the operation the user passes in an array of pointers to communicators. Since communicators are not often created in a contiguous array, pointers to communications should probably be used. The failed_grps is an array of failures in each of those communicators. Some questions: * Should all processes pass in the same set of communicators at all processes? * Should all communicators be duplicates or subsets of one another? * Does this operation run the risk of a circular dependency if the user does not pass in the same set of communicators at all participating processes? Is that something the MPI library should protect the application from? Option B: Implicit inherited validation -------------------- MPI_Comm_validate_many(comm, failed_grp) ---- The idea is to add an additional semantic (or maybe new API) to allows the validation of a communicator to automatically validates all communicators created from it (only dups and subsets of it?). The problem with this is that if an application calls MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of communicators that libraries might be using internally without notification in those libraries. So this breaks the abstraction barrier between the two in possibly a dangerous way. Some questions: * Are there some other semantics that we can add to help protect libraries? (e.g., after implicit validation the first use of the communicator will return a special error code indicating that the communicator has been adjusted). * Are there thread safety issues involved with this? (e.g., the library operates in a concurrent thread with its own duplicate of the communicator. The application does not know about or control the concurrent thread but calls MPI_Comm_validate on its own communicator and implicitly changes the semantics of the duplicate communicator.) * It is only through the call to MPI_Comm_validate that we can provide a uniform group of failed processes globally known. For those that were implicitly validated, do we need to provide a way to access this group after the call? Does this have implications on the amount of storage required for this semantic? Option C: Explicit inherited validation -------------------- MPI_Comm_validate_link(commA, commB); MPI_Comm_validate_many(commA, failed_grp) /* Implies MPI_Comm_validate(commB, NULL) */ MPI_Comm_validate(commA, failed_grp) /* Does not imply MPI_Comm_validate(commB, NULL) */ ---- In this version the application explicitly links communicators. This prevents an application from implicitly altering derived communicators out of their scope (e.g., in use by other libraries). Some questions: * It is only through the call to MPI_Comm_validate that we can provide a uniform group of failed processes globally known. For those that were implicitly validated, do we need to provide a way to access this group after the call (e.g., for commB)? Does this have implications on the amount of storage required for this semantic? * Do we need a mechanism to 'unlink' communicators? Or determine which communicators are linked? * Can a communicator be linked to multiple other communicators? * Is the linking a unidirectional operation? (so in the example above validating commB does not validate commA unless there is a separate MPI_Comm_validate_link(commB, commA) call) Option D: Other -------------------- Something else... Thoughts? -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EDT
by Darius Buntinas 13 Sep '11

13 Sep '11

I'm attaching the latest draft of the ft chapter including the changes to the API that we've been discussing. Sorry for the late notice! -d On Sep 13, 2011, at 3:49 PM, Josh Hursey wrote: > Reminder that we have a teleconf tomorrow, Wednesday 9/14/2011, at Noon EDT. > > Date: September 14, 2011 > Time: Noon EDT/New York > Dial-in information: 877-801-8130 > Code: 1044056 > > Agenda: > - New draft of the Run-Through Stabilization proposal PDF > - Run-Through Stabilization State Management Interface (Darius B., Josh H.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > - Process Recovery (Josh H.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 > - Other items? > > Additional Agenda Items [if time permits]: > - MPI Forum notes discussion (?) > http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php > - Error to Fault changes proposal (Darius B.) > http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php > - Piggybacking? (Martin S.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - Network Channel Failure > http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php > - FT Query Interface > http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php > > > Next Meetings: > * Sept. 14, 2011 (Teleconf) > * Sept. 22, 2011 (MPI Forum Meeting) > * Oct. 5, 2011 (Teleconf) > * Oct. 19, 2011 (Teleconf) > * Oct. 26, 2011 (MPI Forum Meeting) > * Nov. 9, 2011 (Teleconf) > * Nov. 30, 2011 (Teleconf) > > Respond to the list if you have any additional topics that you would > like to discuss. > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ Noon EDT
by Josh Hursey 13 Sep '11

13 Sep '11

Reminder that we have a teleconf tomorrow, Wednesday 9/14/2011, at Noon EDT. Date: September 14, 2011 Time: Noon EDT/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - New draft of the Run-Through Stabilization proposal PDF - Run-Through Stabilization State Management Interface (Darius B., Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - Other items? Additional Agenda Items [if time permits]: - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Piggybacking? (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Sept. 14, 2011 (Teleconf) * Sept. 22, 2011 (MPI Forum Meeting) * Oct. 5, 2011 (Teleconf) * Oct. 19, 2011 (Teleconf) * Oct. 26, 2011 (MPI Forum Meeting) * Nov. 9, 2011 (Teleconf) * Nov. 30, 2011 (Teleconf) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] fault-tolerant collectives
by Bronis R. de Supinski 12 Sep '11

12 Sep '11

It would be useful from a scalability standpoint. I have not thought through how much trouble it might cause from a programmability standpoint. It just seems the inherited mechanism is useful in costs. It might be useful to have an attribute that says to return an error (or some other notification) if it has been validated through inheritance. Then the library fixes could potentially just be local instead of repeating the interprocess work... On Mon, 12 Sep 2011, Darius Buntinas wrote: > Are you're saying that the app might want to validate a library's (internal) communicator for it? > > In general, how safe would it be to do that if the library isn't expecting it? I suppose we could do something like an inherited attribute that makes all subcommunicators automatically validate when it's parent does. > > -d > > On Sep 12, 2011, at 12:12 PM, Bronis R. de Supinski wrote: > >> >> Suppose some communicators are used in libraries >> (duplication of MPI_COMM_WORLD is very common). >> How would any part of the code know about all of >> them? Sure, you could use the profiling interface >> to intercept calls to track them but usual application >> code would not have any record of the handles... >> >> On Mon, 12 Sep 2011, Darius Buntinas wrote: >> >>> >>> On Sep 11, 2011, at 4:45 PM, Josh Hursey wrote: >>> >>>> On Fri, Sep 9, 2011 at 7:57 PM, Graham, Richard L. <rlgraham(a)ornl.gov> wrote: >>>>> I have been talking a reasonable amount with apps folks lately about this proposal, and they first response is often one of shock, as it is not quite what folks initially expect. However, once one explains the background for the proposal, people tend to accept the notions. >>>> >>>> Can you explain a bit more about what they were shocked by? Was it the >>>> general notion of application involved FT, or the interface not being >>>> what they expected/needed? >>>> >>>> >>>>> I agree that we need to define a mechanism for specifying return codes - uniform among surviving ranks, or locally determined types. However, I do believe that we need to add the second set of collectives into 3.0. We have mentioned this as an option for several years (actually since the inception of the group almost 4 years ago), but as a working group never did something explicit about this. There is a reasonable number of apps folks that expect this type of collective communications. >>>> >>>> >>>> It shouldn't be to difficult to specify/add, and a prototype >>>> implementation would be trivial though maybe inefficient at first. Is >>>> this something that we should put in the Stabilization proposal or >>>> bring in as a separate ticket directly afterward? >>>> >>>> Separating the two keeps the initial proposal simpler and users can >>>> get this functionality by wrapping existing collectives in >>>> comm_validate calls (not efficient, but functional). Keeping them >>>> together allows us to address a know interface optimization that >>>> applications want in the first pass. >>> >>> It seems that this can be done in the app itself (or a stand-alone utility library): >>> >>> int MYMPI_Bcast_ft(...) { >>> while (1) { >>> ret = MPI_Bcast(...); >>> if (ret == MPI_ERR_RANK_FAILSTOP) { >>> ret == MPI_Comm_validate(...); >>> if (ret) return ret; >>> } else { >>> return ret; >>> } >>> } >>> } >>> >>> ...or something like that. If an implementation implements it's collectives in a FT manner, then the validate call can be close to a noop (maybe a simple allreduce to make sure the failed_groups are the same). >>> >>>>> One other thing that came up yesterday (I have given 2 talks about the FT stuff in Kobe this week) is that it would be good to be able to specify multiple communicators to mpi_comm_validate(), especially, since a common motif is to dup an existing communicator to isolate communication. This is really the only way that I can think of to avoid un-needed global communication, if more than one communicator is of interest to the app. >>>> >>>> >>>> I've had a couple applications ask about this as well. >>>> >>>> The group has talked about such an interface a few times now, and keep >>>> getting stuck on specifying the interface and semantics of such an >>>> operation. Did they want a function that would take an array of >>>> communicators to validate, or have the validation of one communicator >>>> be inherited by all of the derived communicators? >>>> >>>> I think the array of communicators interface seems like the easiest to >>>> use, and makes it easier to protect libraries. But that leads us to >>>> the question, do all processes (union of processes from all >>>> communicators specified?) have to supply the same set of >>>> communicators? If not, do we run the risk of a circular dependency >>>> causing the call to deadlock? We might be able to pass the >>>> responsibility to avoid such problems off to the user. >>> >>> Yeah, I think it would be easiest to require that the same communicators are specified. And I don't think that would be too difficult for the users to handle. >>> >>> When we talked about "linking" communicators with attributes or something, such that when one communicator is validated, all of the linked communicators are also validated. In that case we said that the all of the linked communicators must be subsets (or dups) of the communicator they're linked to. >>> >>> We could have a similar "subset" restriction for the array-of-communicators-to-comm_validate option. >>> >>>> I'm game for trying to specify this again. The group decided to push >>>> this off to a follow on ticket because it can be achieved (though >>>> inefficiently) by making a call to comm_validate for each of the >>>> communicators, and we had trouble specifying it correctly. So the >>>> question again is should be keep it as a separate ticket or add it to >>>> the stabilization proposal? >>> >>> I'm kinda leaning toward adding it to the current proposal, but I don't want to delay the proposal. >>> -d >>> >>> >>>> Thoughts? >>>> >>>> -- Josh >>>> >>>> >>>>> >>>>> Rich >>>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>> Sent: Friday, September 09, 2011 4:39 PM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] fault-tolerant collectives >>>>> >>>>> >>>>> OK, that makes sense. I'll fix up that text. >>>>> >>>>> Thanks, >>>>> -d >>>>> >>>>> On Sep 9, 2011, at 3:36 PM, Josh Hursey wrote: >>>>> >>>>>> I think that we want to say that an implementation may provide uniform >>>>>> return codes from collectives, but are not required to do so. So this >>>>>> makes then fault tolerant-ish - in the sense that they have to work >>>>>> around failure to return error codes consistently, but not that they >>>>>> finish the collective successfully even if new process failures emerge >>>>>> during the collectives (that would undermine the semantic protections >>>>>> we are putting in place). >>>>>> >>>>>> We should probably not say 'fault tolerant collectives' in the current >>>>>> proposal so we don't confuse things. Maybe 'collectives that provide >>>>>> uniform return codes'? >>>>>> >>>>>> >>>>>> If we want truly fault tolerant collectives (like those described >>>>>> below), then I think we should introduce a different set of functions. >>>>>> The functions should probably return a group of processes that either >>>>>> did or did not participate in creating the final result. Something >>>>>> like: >>>>>> MPI_Reduce_ft(..., &group); >>>>>> >>>>>> I think the true fault tolerant collectives should be left to a follow >>>>>> on ticket since there is a need, but can be easily added as a second >>>>>> step. >>>>>> >>>>>> -- Josh >>>>>> >>>>>> On Fri, Sep 9, 2011 at 4:17 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >>>>>>> >>>>>>> We discussed the option of allowing an implementation to provide fault tolerant (not just fault-aware) collectives. The idea is that even when a process fails, collectives will continue to operate correctly (modulo the failed process). >>>>>>> >>>>>>> Does this imply that the communicator will never become collectively inactive? >>>>>>> >>>>>>> If no, then what's the point of ft collectives? >>>>>>> >>>>>>> If yes, then the application may never get notification that a process has failed and collectives are now running one short. Is this what we really want? >>>>>>> >>>>>>> -d >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Joshua Hursey >>>>>> Postdoctoral Research Associate >>>>>> Oak Ridge National Laboratory >>>>>> hxxp://users.nccs.gov/~jjhursey >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0