September 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] notes from FT plenary and WG sessions for Sept. 2011 mpi forum meeting
by Josh Hursey 23 Sep '11

23 Sep '11

Thanks for the detailed notes. We'll be sure to discuss these issues in the next teleconf, if not on the email list before then. Thanks again, Josh On Fri, Sep 23, 2011 at 5:53 AM, Howard Pritchard <howardp(a)cray.com> wrote: > Hi Folks, > > Here is a summary of my notes from both the plenary session and the > working group. ï¿½ Note that we did the WG before the plenary session. > > Here is a summary of my notes from the working group session - > > The fault tolerant collectives ideas were discussed. ï¿½Darius had > questions about how useful the option for all ranks involved in a coll > op to return the same error code saying the changes to handle this case > vs just calling MPI_Comm_validate after the coll op regardless of the > error code returned. ï¿½Everyone thought it would be good to have more > info from apps people about why this feature - returning uniform error > code - would be much better than the other option. > > There was a lot of discussion of having a 'vector' version of the > MPI_Comm_validate. ï¿½The WG thought it would be sufficient just replace > the existing validate functions with vector versions, with the n=1 case > being equivalent to the existing functionality. > > Feedback from the Eurompi was discussed. Darius talked about a point > someone had raised about what happens with MPI_Comm_split, etc. if an > error occurs and the input communicator is still using errors are fatal > error handler. > > We also discussed what happens when the system can't start up as many > ranks as the user requested on the mpiexec command line. ï¿½Darius pointed > out that the MPI-2 standard already addresses this and said that for > mpich2 now it just tries to start as many ranks as it can. ï¿½It was > decided that how this situation is handled should be described in the > "advice to implementers" in the mpiexec section of the spec. > > First the plenary session - > > Although it is not in the current proposal, Adam Moody brought up the > MPI_Kill functionality again. ï¿½That was discussed and many objections > were raised. ï¿½It is good that we did remove this from the run through > stabilization proposal. > > There was some discussion concerning RMA and how it relates to the > current proposal. ï¿½It was agreed that the FT group needs to sync up with > what the RMA group is doing to make sure there aren't any show-stoppers. > ï¿½Brian Barrett also brought up the need to have an enumeration of the > cases that occur for the existing and proposed RMA synchronization > models and the run through stabilization proposal. > > There was also discussion of the deprecated C++ bindings and how or if > we expect to support fault tolerance when using the C++ bindings given > the way errors are handled when using these bindings. ï¿½It was agreed > that places in the proposal currently reading "return an error" need to > be changed to "raised an error". > > There was a lively discussion of the MPI_ANY_SOURCE issue and the > functionality required to fix up a communicator for pt2pt when a process > has MPI_ANY_SOURCE receives posted. ï¿½Some argued that it may be very > difficult/impossible to support the cancel-like qualities of > MPI_Comm_reenable_any_source. > > George pointed out that we can't just complete-with-error anysource > receives, we'll also need to complete-with-error any receive posted > after the anysource receive that might match the anysource receive. > Consider the example: > > Proc0 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Proc1 > ----- ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ----- > Recv(AS, TAGX)[A] > Recv( 1, TAGX)[B] > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Send(0, TAGX)[C] > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Send(0, TAGX)[D] > > > Without failure, Recv A will match Send C and Recv B will match Send D. > ï¿½If an unrelated process fails, and we only complete-with-error the > anysource, then Recv A will complete with error, and Recv B will match > Send C. ï¿½So we would need to complete-with-error all recvs posted after > an anysource receive with tags that match the tag of the anysource receive. > > This aspect of the proposal definitely needs > reinvestigation/clarification. ï¿½It may also be necessary to discuss this > in more detail with those interested in hardware based mpi tag matching. > > Torsten was not convinced about the feasibility of implementing > MPI_Comm_validate and friends from a theoretical standpoint. ï¿½He asked > if someone has shown whether this is impossible. ï¿½The existing work by > Josh concerning lit. etc may need to be reviewed the next time this > proposal is presented to the forum. > > There was some discussion of the "vector" version of MPI_Comm_validate. > ï¿½The idea of just having a vector of 'comms' didn't seem to go over very > well. ï¿½What seemed to be more palatable would be to have a first comm > argument, followed by a vector of comms which are derived from the first > comm. > > Somehow we returned to looking at examples like the MPI_Bcast example in > the proposal. ï¿½The early breakout from the loop caused very animated > discussion, but no real consensus about what to do about this. > > Aspects of communicator creation was also brought up. ï¿½A suggestion was > given that for routines that create communicators, and for which the > default errors are fatal errors are fatal error handle is associated > with the input comm, that rather than immediately returning an error if > the operation fails, the app would be allowed to attach a non-default > error handler which would then raise the error, much like is documented > for MPI_Init. > > > Howard > -- > Howard Pritchard > Software Engineering > Cray, Inc. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] notes from FT plenary and WG sessions for Sept. 2011 mpi forum meeting
by Howard Pritchard 23 Sep '11

23 Sep '11

Hi Folks, Here is a summary of my notes from both the plenary session and the working group. Note that we did the WG before the plenary session. Here is a summary of my notes from the working group session - The fault tolerant collectives ideas were discussed. Darius had questions about how useful the option for all ranks involved in a coll op to return the same error code saying the changes to handle this case vs just calling MPI_Comm_validate after the coll op regardless of the error code returned. Everyone thought it would be good to have more info from apps people about why this feature - returning uniform error code - would be much better than the other option. There was a lot of discussion of having a 'vector' version of the MPI_Comm_validate. The WG thought it would be sufficient just replace the existing validate functions with vector versions, with the n=1 case being equivalent to the existing functionality. Feedback from the Eurompi was discussed. Darius talked about a point someone had raised about what happens with MPI_Comm_split, etc. if an error occurs and the input communicator is still using errors are fatal error handler. We also discussed what happens when the system can't start up as many ranks as the user requested on the mpiexec command line. Darius pointed out that the MPI-2 standard already addresses this and said that for mpich2 now it just tries to start as many ranks as it can. It was decided that how this situation is handled should be described in the "advice to implementers" in the mpiexec section of the spec. First the plenary session - Although it is not in the current proposal, Adam Moody brought up the MPI_Kill functionality again. That was discussed and many objections were raised. It is good that we did remove this from the run through stabilization proposal. There was some discussion concerning RMA and how it relates to the current proposal. It was agreed that the FT group needs to sync up with what the RMA group is doing to make sure there aren't any show-stoppers. Brian Barrett also brought up the need to have an enumeration of the cases that occur for the existing and proposed RMA synchronization models and the run through stabilization proposal. There was also discussion of the deprecated C++ bindings and how or if we expect to support fault tolerance when using the C++ bindings given the way errors are handled when using these bindings. It was agreed that places in the proposal currently reading "return an error" need to be changed to "raised an error". There was a lively discussion of the MPI_ANY_SOURCE issue and the functionality required to fix up a communicator for pt2pt when a process has MPI_ANY_SOURCE receives posted. Some argued that it may be very difficult/impossible to support the cancel-like qualities of MPI_Comm_reenable_any_source. George pointed out that we can't just complete-with-error anysource receives, we'll also need to complete-with-error any receive posted after the anysource receive that might match the anysource receive. Consider the example: Proc0 Proc1 ----- ----- Recv(AS, TAGX)[A] Recv( 1, TAGX)[B] Send(0, TAGX)[C] Send(0, TAGX)[D] Without failure, Recv A will match Send C and Recv B will match Send D. If an unrelated process fails, and we only complete-with-error the anysource, then Recv A will complete with error, and Recv B will match Send C. So we would need to complete-with-error all recvs posted after an anysource receive with tags that match the tag of the anysource receive. This aspect of the proposal definitely needs reinvestigation/clarification. It may also be necessary to discuss this in more detail with those interested in hardware based mpi tag matching. Torsten was not convinced about the feasibility of implementing MPI_Comm_validate and friends from a theoretical standpoint. He asked if someone has shown whether this is impossible. The existing work by Josh concerning lit. etc may need to be reviewed the next time this proposal is presented to the forum. There was some discussion of the "vector" version of MPI_Comm_validate. The idea of just having a vector of 'comms' didn't seem to go over very well. What seemed to be more palatable would be to have a first comm argument, followed by a vector of comms which are derived from the first comm. Somehow we returned to looking at examples like the MPI_Bcast example in the proposal. The early breakout from the loop caused very animated discussion, but no real consensus about what to do about this. Aspects of communicator creation was also brought up. A suggestion was given that for routines that create communicators, and for which the default errors are fatal errors are fatal error handle is associated with the input comm, that rather than immediately returning an error if the operation fails, the app would be allowed to attach a non-default error handler which would then raise the error, much like is documented for MPI_Init. Howard -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-ft] MPI Forum Feedback?
by Graham, Richard L. 22 Sep '11

22 Sep '11

Howard took detailed notes, and will type them up tonight. Rich On Sep 22, 2011, at 1:40 PM, Josh Hursey wrote: > For those attending the MPI Forum meeting in Greece today. How did the > working group and plenary sessions go? Did we get any specific > feedback on the proposal? > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > hxxp://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

[Mpi3-ft] MPI Forum Feedback?
by Josh Hursey 22 Sep '11

22 Sep '11

For those attending the MPI Forum meeting in Greece today. How did the working group and plenary sessions go? Did we get any specific feedback on the proposal? Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Graham, Richard L. 19 Sep '11

19 Sep '11

Bronis, I have seen you raise the memory issue several times. From my perspective the memory issue is an implementation issue, and is no different than that of the communication implementation itself. The current proposal is such that an efficient implementation can be developed - I am not aware of any interface issues that require non-scalable memory usage. From my perspective, the reason to validate multiple communicators are - user convenience (weak) - reduce global communications. Unfortunately, since one is quarrying for the state of a global object (the communicator, file handle, window) local information is not sufficient to infer it's state. Rich On Sep 15, 2011, at 6:08 PM, Bronis R. de Supinski wrote: > >> From a scalability standpoint and from a MPI implementation > memory usage standpoint, you want validating any communicator > that includes an dead endpoint to eliminate ever having to > use collective communication to validate other communicators > that include that endpoint. If you can achieve that by the > current interface then you do not need to add anything (the > validate becomes a local call so no big deal, it just ensures > the code that uses that communicator has current information). > > Otherwise, you want validating a communicator that includes > that endpoint to validate all other communicators that use > that endpoint, regardless of how the communicators have been > derived. You do not want a inherited interface, either explicit > or implicit (explicit is too hard to use and does not solve the > real problem of excessive cost being forced on the user by > the interface; similarly implicit but only inherited does not > solve the problem). The key is to provide some mechanism to > alert the user that the communicator has been backdoor validated. > > > > On Thu, 15 Sep 2011, Josh Hursey wrote: > >> On Wed, Sep 14, 2011 at 4:17 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: >>> Hi Josh, >>> >>>> >>>> Workaround: >>>> -------------------- >>>> Call MPI_Comm_validate over all of the communicators individually. >>>> This would involve 'num_comms' collective operations and likely impede >>>> scalability. >>>> >>>> for(i=0; i < num_comms; ++i) { >>>> MPI_Comm_validate(comm[i], failed_grps[i]); >>>> } >>>> >>> >>> Would it not be possible for the app to create a communicator that is the union of all the processes, and subsequently call validate only on that 'super' communicator? I hope I am not missing something from your example. >> >> In the current spec, no. MPI_Comm_validate only changes the state of >> the communicator passed. We probably want to create a new API like >> MPI_Comm_validate_many() to host these new semantics. >> >> It is important to remember that the validate operation changes the >> communicator (primarily just the 'are_collectives_enabled' flag on the >> communicator), and not anything to do with elements of the group that >> form it. >> >> Currently after creation, a communicator does not need to track from >> which communicators it was created (at least that's the way I >> understand it). So creating a super communicator and calling >> MPI_Comm_validate_many() on that would require such tracking to have >> the validation propagate to all of the communicators that built it. So >> we could do it, but the additional state tracking would force >> additional memory consumption even if the operation is never used, >> which is slightly problematic. >> >> >>> >>> I liked your Option B as such, however, as you point out, it has significant problems in case of applications consisting of several layers of libraries. >> >> >> Thinking through your question above, I think Option B would require >> that we track the heritage of communicators after creation, which >> would increase memory consumption. It would also require us to >> maintain that linkage across communicator destruction. For example, >> ----------- >> MPI_Comm_dup(MPI_COMM_WORLD, commA); >> // MCW is linked to commA >> MPI_Comm_dup(commA, commB); >> // MCW is linked to commA >> // commA is linked to commB >> MPI_Comm_dup(commB, commC); >> // MCW is linked to commA >> // commA is linked to commB >> // commB is linked to commC >> MPI_Comm_free(commB); >> // MCW is linked to commA >> // commA is linked to commC (since commB is now gone) >> ------------ >> >> In the discussion so far it seems that the inheritance is only one >> way. Meaning that in the example above calling >> MPI_Comm_validate_many() on commA would validate commC (and commB if >> it is still around), but not MPI_COMM_WORLD. Is that what we are >> looking for, or do we want it to be more complete? >> >> >> The explicit linking in Option C puts the user in more control over >> the overhead of tracking connections between communicators, but has >> other issues. :/ >> >> -- Josh >> >> >>> >>> Sayantan. >>> >>>> >>>> Option A: >>>> Array of communicators >>>> -------------------- >>>> MPI_Comm_validate_many(comm[], num_comms, failed_grp) >>>> Validate 'num_comms' communicators, and return a failed group. >>>> - or - >>>> MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) >>>> Validate 'num_comms' communicators, and return a failed group for each >>>> communicator. >>>> ---- >>>> >>>> In this version of the operation the user passes in an array of >>>> pointers to communicators. Since communicators are not often created >>>> in a contiguous array, pointers to communications should probably be >>>> used. The failed_grps is an array of failures in each of those >>>> communicators. >>>> >>>> Some questions: >>>> * Should all processes pass in the same set of communicators at all >>>> processes? >>>> * Should all communicators be duplicates or subsets of one another? >>>> * Does this operation run the risk of a circular dependency if the >>>> user does not pass in the same set of communicators at all >>>> participating processes? Is that something the MPI library should >>>> protect the application from? >>>> >>>> >>>> Option B: >>>> Implicit inherited validation >>>> -------------------- >>>> MPI_Comm_validate_many(comm, failed_grp) >>>> ---- >>>> >>>> The idea is to add an additional semantic (or maybe new API) to allows >>>> the validation of a communicator to automatically validates all >>>> communicators created from it (only dups and subsets of it?). >>>> >>>> The problem with this is that if an application calls >>>> MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of >>>> communicators that libraries might be using internally without >>>> notification in those libraries. So this breaks the abstraction >>>> barrier between the two in possibly a dangerous way. >>>> >>>> Some questions: >>>> * Are there some other semantics that we can add to help protect >>>> libraries? (e.g., after implicit validation the first use of the >>>> communicator will return a special error code indicating that the >>>> communicator has been adjusted). >>>> * Are there thread safety issues involved with this? (e.g., the >>>> library operates in a concurrent thread with its own duplicate of the >>>> communicator. The application does not know about or control the >>>> concurrent thread but calls MPI_Comm_validate on its own communicator >>>> and implicitly changes the semantics of the duplicate communicator.) >>>> * It is only through the call to MPI_Comm_validate that we can >>>> provide a uniform group of failed processes globally known. For those >>>> that were implicitly validated, do we need to provide a way to access >>>> this group after the call? Does this have implications on the amount >>>> of storage required for this semantic? >>>> >>>> >>>> Option C: >>>> Explicit inherited validation >>>> -------------------- >>>> MPI_Comm_validate_link(commA, commB); >>>> MPI_Comm_validate_many(commA, failed_grp) >>>> /* Implies MPI_Comm_validate(commB, NULL) */ >>>> >>>> MPI_Comm_validate(commA, failed_grp) >>>> /* Does not imply MPI_Comm_validate(commB, NULL) */ >>>> ---- >>>> >>>> In this version the application explicitly links communicators. This >>>> prevents an application from implicitly altering derived communicators >>>> out of their scope (e.g., in use by other libraries). >>>> >>>> Some questions: >>>> * It is only through the call to MPI_Comm_validate that we can >>>> provide a uniform group of failed processes globally known. For those >>>> that were implicitly validated, do we need to provide a way to access >>>> this group after the call (e.g., for commB)? Does this have >>>> implications on the amount of storage required for this semantic? >>>> * Do we need a mechanism to 'unlink' communicators? Or determine >>>> which communicators are linked? >>>> * Can a communicator be linked to multiple other communicators? >>>> * Is the linking a unidirectional operation? (so in the example above >>>> validating commB does not validate commA unless there is a separate >>>> MPI_Comm_validate_link(commB, commA) call) >>>> >>>> >>>> Option D: >>>> Other >>>> -------------------- >>>> Something else... >>>> >>>> >>>> Thoughts? >>>> >>>> -- Josh >>>> >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> hxxp://users.nccs.gov/~jjhursey >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> hxxp://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] fault-tolerant collectives
by Graham, Richard L. 19 Sep '11

19 Sep '11

Just catching up here .... On Sep 12, 2011, at 1:40 PM, Bronis R. de Supinski wrote: > > It would be useful from a scalability standpoint. > I have not thought through how much trouble it > might cause from a programmability standpoint. > It just seems the inherited mechanism is useful > in costs. It might be useful to have an attribute > that says to return an error (or some other notification) > if it has been validated through inheritance. Then > the library fixes could potentially just be local > instead of repeating the interprocess work... I believe that inheritance could be dangerous, and is why I suggested a list of communicators. The origin of the validate call is the desire by apps to be able to have a place to check on the state of the binary at the stage that the call is made, so if an app has the inheritance properties wrong, they are out of luck. Listing the ones of interest is a bit less risky. On the other hand, I suppose we can just make it the user's responsibility to "get it right". > > On Mon, 12 Sep 2011, Darius Buntinas wrote: > >> Are you're saying that the app might want to validate a library's (internal) communicator for it? >> >> In general, how safe would it be to do that if the library isn't expecting it? I suppose we could do something like an inherited attribute that makes all subcommunicators automatically validate when it's parent does. >> >> -d >> >> On Sep 12, 2011, at 12:12 PM, Bronis R. de Supinski wrote: >> >>> >>> Suppose some communicators are used in libraries >>> (duplication of MPI_COMM_WORLD is very common). >>> How would any part of the code know about all of >>> them? Sure, you could use the profiling interface >>> to intercept calls to track them but usual application >>> code would not have any record of the handles... >>> >>> On Mon, 12 Sep 2011, Darius Buntinas wrote: >>> >>>> >>>> On Sep 11, 2011, at 4:45 PM, Josh Hursey wrote: >>>> >>>>> On Fri, Sep 9, 2011 at 7:57 PM, Graham, Richard L. <rlgraham(a)ornl.gov> wrote: >>>>>> I have been talking a reasonable amount with apps folks lately about this proposal, and they first response is often one of shock, as it is not quite what folks initially expect. However, once one explains the background for the proposal, people tend to accept the notions. >>>>> >>>>> Can you explain a bit more about what they were shocked by? Was it the >>>>> general notion of application involved FT, or the interface not being >>>>> what they expected/needed? [rich] The notion of different return codes for different processes is what really surprises them. This is no different that things are today, with the exception that the errors_abort code which is used basically 100% of the time takes care of this. Once the motivation to minimize the performance penalty, under the assumption of a low error rate is explained, this makes sense. >>>>> >>>>> >>>>>> I agree that we need to define a mechanism for specifying return codes - uniform among surviving ranks, or locally determined types. However, I do believe that we need to add the second set of collectives into 3.0. We have mentioned this as an option for several years (actually since the inception of the group almost 4 years ago), but as a working group never did something explicit about this. There is a reasonable number of apps folks that expect this type of collective communications. >>>>> >>>>> >>>>> It shouldn't be to difficult to specify/add, and a prototype >>>>> implementation would be trivial though maybe inefficient at first. Is >>>>> this something that we should put in the Stabilization proposal or >>>>> bring in as a separate ticket directly afterward? [rich] The proposal needs to be sufficiently broad scoped, that it satisfies a wide range of apps, so this does need to go into the first version. However, this is not new - these sort of collectives have been implemented before, so I view these as a small change - only adding some sort of flag indicating the type of collectives to be used at run-time. >>>>> >>>>> Separating the two keeps the initial proposal simpler and users can >>>>> get this functionality by wrapping existing collectives in >>>>> comm_validate calls (not efficient, but functional). Keeping them >>>>> together allows us to address a know interface optimization that >>>>> applications want in the first pass. >>>> >>>> It seems that this can be done in the app itself (or a stand-alone utility library) [rich] this is true about all collectives. The apps can implement thees on their own. The reason for including them in the standard is that they are used a lot, and are very expensive to optimize, so having library implementations that do this for all apps is beneficial here. A similar argument holds here for the collectives. >>>> >>>> int MYMPI_Bcast_ft(...) { >>>> while (1) { >>>> ret = MPI_Bcast(...); >>>> if (ret == MPI_ERR_RANK_FAILSTOP) { >>>> ret == MPI_Comm_validate(...); >>>> if (ret) return ret; >>>> } else { >>>> return ret; >>>> } >>>> } >>>> } >>>> >>>> ...or something like that. If an implementation implements it's collectives in a FT manner, then the validate call can be close to a noop (maybe a simple allreduce to make sure the failed_groups are the same). >>>> >>>>>> One other thing that came up yesterday (I have given 2 talks about the FT stuff in Kobe this week) is that it would be good to be able to specify multiple communicators to mpi_comm_validate(), especially, since a common motif is to dup an existing communicator to isolate communication. This is really the only way that I can think of to avoid un-needed global communication, if more than one communicator is of interest to the app. >>>>> >>>>> >>>>> I've had a couple applications ask about this as well. >>>>> >>>>> The group has talked about such an interface a few times now, and keep >>>>> getting stuck on specifying the interface and semantics of such an >>>>> operation. Did they want a function that would take an array of >>>>> communicators to validate, or have the validation of one communicator >>>>> be inherited by all of the derived communicators? [rich] yes, I would just extend the current definition to take in an array of communicators, and return an array of group handles. >>>>> >>>>> I think the array of communicators interface seems like the easiest to >>>>> use, and makes it easier to protect libraries. But that leads us to >>>>> the question, do all processes (union of processes from all >>>>> communicators specified?) have to supply the same set of >>>>> communicators? If not, do we run the risk of a circular dependency >>>>> causing the call to deadlock? We might be able to pass the >>>>> responsibility to avoid such problems off to the user. >>>> >>>> Yeah, I think it would be easiest to require that the same communicators are specified. And I don't think that would be too difficult for the users to handle. >>>> >>>> When we talked about "linking" communicators with attributes or something, such that when one communicator is validated, all of the linked communicators are also validated. In that case we said that the all of the linked communicators must be subsets (or dups) of the communicator they're linked to. >>>> >>>> We could have a similar "subset" restriction for the array-of-communicators-to-comm_validate option. [rich] I don't think there is an issue here with deadlock, and there is no need to restrict the list of communicators. Just like any collective communication, all are required to call these in the same order - validate is no different. If the library can detect some optimization opportunities, this is great, otherwise we are no worse off than having a series of independent calls. Rich >>>> >>>>> I'm game for trying to specify this again. The group decided to push >>>>> this off to a follow on ticket because it can be achieved (though >>>>> inefficiently) by making a call to comm_validate for each of the >>>>> communicators, and we had trouble specifying it correctly. So the >>>>> question again is should be keep it as a separate ticket or add it to >>>>> the stabilization proposal? >>>> >>>> I'm kinda leaning toward adding it to the current proposal, but I don't want to delay the proposal. >>>> -d >>>> >>>> >>>>> Thoughts? >>>>> >>>>> -- Josh >>>>> >>>>> >>>>>> >>>>>> Rich >>>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>>> Sent: Friday, September 09, 2011 4:39 PM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>> Subject: Re: [Mpi3-ft] fault-tolerant collectives >>>>>> >>>>>> >>>>>> OK, that makes sense. I'll fix up that text. >>>>>> >>>>>> Thanks, >>>>>> -d >>>>>> >>>>>> On Sep 9, 2011, at 3:36 PM, Josh Hursey wrote: >>>>>> >>>>>>> I think that we want to say that an implementation may provide uniform >>>>>>> return codes from collectives, but are not required to do so. So this >>>>>>> makes then fault tolerant-ish - in the sense that they have to work >>>>>>> around failure to return error codes consistently, but not that they >>>>>>> finish the collective successfully even if new process failures emerge >>>>>>> during the collectives (that would undermine the semantic protections >>>>>>> we are putting in place). >>>>>>> >>>>>>> We should probably not say 'fault tolerant collectives' in the current >>>>>>> proposal so we don't confuse things. Maybe 'collectives that provide >>>>>>> uniform return codes'? >>>>>>> >>>>>>> >>>>>>> If we want truly fault tolerant collectives (like those described >>>>>>> below), then I think we should introduce a different set of functions. >>>>>>> The functions should probably return a group of processes that either >>>>>>> did or did not participate in creating the final result. Something >>>>>>> like: >>>>>>> MPI_Reduce_ft(..., &group); >>>>>>> >>>>>>> I think the true fault tolerant collectives should be left to a follow >>>>>>> on ticket since there is a need, but can be easily added as a second >>>>>>> step. >>>>>>> >>>>>>> -- Josh >>>>>>> >>>>>>> On Fri, Sep 9, 2011 at 4:17 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >>>>>>>> >>>>>>>> We discussed the option of allowing an implementation to provide fault tolerant (not just fault-aware) collectives. The idea is that even when a process fails, collectives will continue to operate correctly (modulo the failed process). >>>>>>>> >>>>>>>> Does this imply that the communicator will never become collectively inactive? >>>>>>>> >>>>>>>> If no, then what's the point of ft collectives? >>>>>>>> >>>>>>>> If yes, then the application may never get notification that a process has failed and collectives are now running one short. Is this what we really want? >>>>>>>> >>>>>>>> -d >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Joshua Hursey >>>>>>> Postdoctoral Research Associate >>>>>>> Oak Ridge National Laboratory >>>>>>> hxxp://users.nccs.gov/~jjhursey >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Joshua Hursey >>>>> Postdoctoral Research Associate >>>>> Oak Ridge National Laboratory >>>>> hxxp://users.nccs.gov/~jjhursey >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Updates to ft chapter
by Josh Hursey 16 Sep '11

16 Sep '11

I updated the document with the following changes: Change Log * Minor wording updates, and a few movements - per diff sent to the list (this thread). http://lists.mpi-forum.org/mpi3-ft/2011/09/0850.php * 17.1: Slight rewording of advice to implementors text (thanks to Sayantan and Darius) * 17.4.4: Fixed typo in Example 17.4 (thanks to Sayantan) * 17.4.4: Combined examples 17.2 and 17.3. (thanks to Sayantan) https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… I did -not- change any of the bigger items in the list below. -- Josh On Thu, Sep 15, 2011 at 1:27 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > I made some minor-ish edits. I attached the diff to this email for > review. Feel free to commit it if you think it is good to go. > > Some larger items that I did not want to change/adjust before > discussing with the group: > > 17.1: > ï¿½Does the Advice to Implementors buy us anything? Should it be reworded? > > 17.2: > ï¿½Do we need the definitions of 'error' and 'failure'? We don't rely on > these definitions in the text beyond their previously implied > definitions in the standard. If we do not need them, then it might be > good to drop them to reduce complexity. > > 17.5: > ï¿½It was suggested that we try to clarify this paragraph with the > Rationale. Any suggestions? > > 17.5.2: > ï¿½Should we go ahead and pull the Advice to implementors regarding the > return value of mpiexec into a separate ticket? Or should we keep it > in the document and pull it if we get pushback? (I think on the call > we decided the latter, but I forget now). > > 17.7: > In the second Rationale paragraph. I moved the first sentence to > 17.7.1. But I think we can drop the rest of the rationale. I do not > know if it is terribly instructive. > > 17.7.1: > I updated the rational to account for the MPI_Reduce numerical > stability recommendation. > --------------- > Rationale. The MPI_COMM_VALIDATE and MPI_ICOMM_VALIDATE operations > provide the MPI implementation an opportunity to restructure > collective communication patterns before the communicator is used by > the alive process. This may allow for improved collective performance > after process failure. It should be noted such optimizations might > change the consistency recommendation for MPI_REDUCE in the advice to > implementors in Section ??. It is strongly recommended that the > consistency recommendation hold for MPI_REDUCE between consecutive > collective activations of a communicator using a collective validation > operation (e.g, MPI_COMM_VALIDATE). (End of rationale.) > --------------- > > 17.8.2: > Note that I moved the Advice to users regarding libraries to here, per > the teleconf. > > 17.12.1: > ï¿½Added back the Advice to users regarding the 'sync-barrier-sync' > semantic for MPI_File_validate. > > > Thanks, > Josh > > On Wed, Sep 14, 2011 at 1:58 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> >> I've made some changes we discussed on the phone this morning. ï¿½You can find the latest pdf here (or at the bottom of the "Modified run-through stabilization" page on the wiki): >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… >> >> Here's a summary of the changes: >> ï¿½* Changed MPI_ERR_RANK_FAIL_STOP to MPI_ERR_PROC_FAIL_STOP (because a "rank" doesn't fail, a "processes" does) >> ï¿½* Fixed up usage of rank vs process in the chapter. >> ï¿½* Removed MPI_COMM_COLLECTIVES_ENABLED function because it returns local version of a global state which is meaningless for applications. >> ï¿½* Added MPI_COMM_ANY_SOURCE_ENABLED >> ï¿½* Moved the definition of MPI_COMM_VALIDATE et.al. earlier in the section, and added a new subsection. >> >> Please look over my changes, especially how I rearranged the collectives section for the definition of MPI_COMM_VALIDATE, and let me know if they look OK. >> >> Thanks! >> -d >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Josh Hursey 16 Sep '11

16 Sep '11

On Thu, Sep 15, 2011 at 6:08 PM, Bronis R. de Supinski <bronis(a)llnl.gov> wrote: > >> From a scalability standpoint and from a MPI implementation > > memory usage standpoint, you want validating any communicator > that includes an dead endpoint to eliminate ever having to > use collective communication to validate other communicators > that include that endpoint. If you can achieve that by the > current interface then you do not need to add anything (the > validate becomes a local call so no big deal, it just ensures > the code that uses that communicator has current information). Note that if you do not want to use collectives, just point-to-point then you may never have to call MPI_Comm_validate at all - it is really just for re-enabling collectives and providing a consistent global view of the failure set. For collective communication, I don't think we can achieve the local semantic with the current version of MPI_Comm_validate() so we are definitely talking about a new interface. The synchronization and consistency aspects of MPI_Comm_validate() are pretty important as building blocks for many fault tolerant applications. Termination protocols often require that at some point each process knows that everyone else agrees upon the same state, so that they can take consistent action. Without faults you can infer this behavior based on messages received and a single context of execution. But with faults occurring nondeterministically, and notification at each process happening at different points in the execution, it can be difficult to coordinate recovery with other processes. We could think about what a MPI_Comm_validate_local() version might look like. Something that validates a communicator locally with respect to say another communicator. -------------- MPI_Comm_validate_many(MPI_COMM_WORLD, &grp); // This implicitly validates all other communicators containing // any of the failed processes in &grp. Collective across MCW MPI_Comm_validate_local(commA, MPI_COMM_WORLD, &grp); // Locally validate commA based on the last validated state of MCW -------------- So it would re-enable collectives locally and return a list of failed processes with respect to the last MPI_Comm_validate (humm we might have to store this information on the communicator which we have avoided so far). Would we run into cross-process consistency issues if one process validated_local and started using collectives on commA before another process called validate_local? This might be something interesting to think through as another alternative. The problem with the MPI_Comm_validate_local call is that you need to know which communicator was validated to know which to validate locally against (gets into the user complexity trap). > > Otherwise, you want validating a communicator that includes > that endpoint to validate all other communicators that use > that endpoint, regardless of how the communicators have been > derived. You do not want a inherited interface, either explicit > or implicit (explicit is too hard to use and does not solve the > real problem of excessive cost being forced on the user by > the interface; similarly implicit but only inherited does not > solve the problem). The key is to provide some mechanism to > alert the user that the communicator has been backdoor validated. > So if the application calls MPI_Comm_validate_many(MPI_COMM_WORLD) then all other communicators containing the identified failed ranks would automatically be validated in the same step. We could return a special error code MPI_ERR_YOU_HAVE_BEEN_VALIDATED on the first use of the derived communicator after the validate call completes (maybe with some protection for point-to-point operations to non-failed processes). This would warn the application that after this point collectives are re-enabled for this derived communicator, and provide the notification of being backdoor validated. The application/library would be forced to check the return code from all MPI calls for this special error code if it uses collective operations. I'm not sure what we might need to say about the state of operations on other communicators at the time the MPI_Comm_validate_many() call is active or the concurrent use of MPI_Comm_validate_many() in multiple communicators. Are there any threading issues between a library working in a thread calling collectives and the main application periodically calling MPI_Comm_validate_many()? What do folks think? -- Josh > > > On Thu, 15 Sep 2011, Josh Hursey wrote: > >> On Wed, Sep 14, 2011 at 4:17 PM, Sur, Sayantan <sayantan.sur(a)intel.com> >> wrote: >>> >>> Hi Josh, >>> >>>> >>>> Workaround: >>>> -------------------- >>>> Call MPI_Comm_validate over all of the communicators individually. >>>> This would involve 'num_comms' collective operations and likely impede >>>> scalability. >>>> >>>> for(i=0; i < num_comms; ++i) { >>>> ï¿½ MPI_Comm_validate(comm[i], failed_grps[i]); >>>> } >>>> >>> >>> Would it not be possible for the app to create a communicator that is the >>> union of all the processes, and subsequently call validate only on that >>> 'super' communicator? I hope I am not missing something from your example. >> >> In the current spec, no. MPI_Comm_validate only changes the state of >> the communicator passed. We probably want to create a new API like >> MPI_Comm_validate_many() to host these new semantics. >> >> It is important to remember that the validate operation changes the >> communicator (primarily just the 'are_collectives_enabled' flag on the >> communicator), and not anything to do with elements of the group that >> form it. >> >> Currently after creation, a communicator does not need to track from >> which communicators it was created (at least that's the way I >> understand it). So creating a super communicator and calling >> MPI_Comm_validate_many() on that would require such tracking to have >> the validation propagate to all of the communicators that built it. So >> we could do it, but the additional state tracking would force >> additional memory consumption even if the operation is never used, >> which is slightly problematic. >> >> >>> >>> I liked your Option B as such, however, as you point out, it has >>> significant problems in case of applications consisting of several layers of >>> libraries. >> >> >> Thinking through your question above, I think Option B would require >> that we track the heritage of communicators after creation, which >> would increase memory consumption. It would also require us to >> maintain that linkage across communicator destruction. For example, >> ----------- >> MPI_Comm_dup(MPI_COMM_WORLD, commA); >> // MCW ï¿½ is linked to commA >> MPI_Comm_dup(commA, commB); >> // MCW ï¿½ is linked to commA >> // commA is linked to commB >> MPI_Comm_dup(commB, commC); >> // MCW ï¿½ is linked to commA >> // commA is linked to commB >> // commB is linked to commC >> MPI_Comm_free(commB); >> // MCW ï¿½ is linked to commA >> // commA is linked to commC (since commB is now gone) >> ------------ >> >> In the discussion so far it seems that the inheritance is only one >> way. Meaning that in the example above calling >> MPI_Comm_validate_many() on commA would validate commC (and commB if >> it is still around), but not MPI_COMM_WORLD. Is that what we are >> looking for, or do we want it to be more complete? >> >> >> The explicit linking in Option C puts the user in more control over >> the overhead of tracking connections between communicators, but has >> other issues. :/ >> >> -- Josh >> >> >>> >>> Sayantan. >>> >>>> >>>> Option A: >>>> Array of communicators >>>> -------------------- >>>> MPI_Comm_validate_many(comm[], num_comms, failed_grp) >>>> Validate 'num_comms' communicators, and return a failed group. >>>> ï¿½ - or - >>>> MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) >>>> Validate 'num_comms' communicators, and return a failed group for each >>>> communicator. >>>> ---- >>>> >>>> In this version of the operation the user passes in an array of >>>> pointers to communicators. Since communicators are not often created >>>> in a contiguous array, pointers to communications should probably be >>>> used. The failed_grps is an array of failures in each of those >>>> communicators. >>>> >>>> Some questions: >>>> ï¿½* Should all processes pass in the same set of communicators at all >>>> processes? >>>> ï¿½* Should all communicators be duplicates or subsets of one another? >>>> ï¿½* Does this operation run the risk of a circular dependency if the >>>> user does not pass in the same set of communicators at all >>>> participating processes? Is that something the MPI library should >>>> protect the application from? >>>> >>>> >>>> Option B: >>>> Implicit inherited validation >>>> -------------------- >>>> MPI_Comm_validate_many(comm, failed_grp) >>>> ---- >>>> >>>> The idea is to add an additional semantic (or maybe new API) to allows >>>> the validation of a communicator to automatically validates all >>>> communicators created from it (only dups and subsets of it?). >>>> >>>> The problem with this is that if an application calls >>>> MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of >>>> communicators that libraries might be using internally without >>>> notification in those libraries. So this breaks the abstraction >>>> barrier between the two in possibly a dangerous way. >>>> >>>> Some questions: >>>> ï¿½* Are there some other semantics that we can add to help protect >>>> libraries? (e.g., after implicit validation the first use of the >>>> communicator will return a special error code indicating that the >>>> communicator has been adjusted). >>>> ï¿½* Are there thread safety issues involved with this? (e.g., the >>>> library operates in a concurrent thread with its own duplicate of the >>>> communicator. The application does not know about or control the >>>> concurrent thread but calls MPI_Comm_validate on its own communicator >>>> and implicitly changes the semantics of the duplicate communicator.) >>>> ï¿½* It is only through the call to MPI_Comm_validate that we can >>>> provide a uniform group of failed processes globally known. For those >>>> that were implicitly validated, do we need to provide a way to access >>>> this group after the call? Does this have implications on the amount >>>> of storage required for this semantic? >>>> >>>> >>>> Option C: >>>> Explicit inherited validation >>>> -------------------- >>>> MPI_Comm_validate_link(commA, commB); >>>> MPI_Comm_validate_many(commA, failed_grp) >>>> /* Implies MPI_Comm_validate(commB, NULL) */ >>>> >>>> MPI_Comm_validate(commA, failed_grp) >>>> /* Does not imply MPI_Comm_validate(commB, NULL) */ >>>> ---- >>>> >>>> In this version the application explicitly links communicators. This >>>> prevents an application from implicitly altering derived communicators >>>> out of their scope (e.g., in use by other libraries). >>>> >>>> Some questions: >>>> ï¿½* It is only through the call to MPI_Comm_validate that we can >>>> provide a uniform group of failed processes globally known. For those >>>> that were implicitly validated, do we need to provide a way to access >>>> this group after the call (e.g., for commB)? Does this have >>>> implications on the amount of storage required for this semantic? >>>> ï¿½* Do we need a mechanism to 'unlink' communicators? Or determine >>>> which communicators are linked? >>>> ï¿½* Can a communicator be linked to multiple other communicators? >>>> ï¿½* Is the linking a unidirectional operation? (so in the example above >>>> validating commB does not validate commA unless there is a separate >>>> MPI_Comm_validate_link(commB, commA) call) >>>> >>>> >>>> Option D: >>>> Other >>>> -------------------- >>>> Something else... >>>> >>>> >>>> Thoughts? >>>> >>>> -- Josh >>>> >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Bronis R. de Supinski 15 Sep '11

15 Sep '11

>From a scalability standpoint and from a MPI implementation memory usage standpoint, you want validating any communicator that includes an dead endpoint to eliminate ever having to use collective communication to validate other communicators that include that endpoint. If you can achieve that by the current interface then you do not need to add anything (the validate becomes a local call so no big deal, it just ensures the code that uses that communicator has current information). Otherwise, you want validating a communicator that includes that endpoint to validate all other communicators that use that endpoint, regardless of how the communicators have been derived. You do not want a inherited interface, either explicit or implicit (explicit is too hard to use and does not solve the real problem of excessive cost being forced on the user by the interface; similarly implicit but only inherited does not solve the problem). The key is to provide some mechanism to alert the user that the communicator has been backdoor validated. On Thu, 15 Sep 2011, Josh Hursey wrote: > On Wed, Sep 14, 2011 at 4:17 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: >> Hi Josh, >> >>> >>> Workaround: >>> -------------------- >>> Call MPI_Comm_validate over all of the communicators individually. >>> This would involve 'num_comms' collective operations and likely impede >>> scalability. >>> >>> for(i=0; i < num_comms; ++i) { >>> ï¿½ MPI_Comm_validate(comm[i], failed_grps[i]); >>> } >>> >> >> Would it not be possible for the app to create a communicator that is the union of all the processes, and subsequently call validate only on that 'super' communicator? I hope I am not missing something from your example. > > In the current spec, no. MPI_Comm_validate only changes the state of > the communicator passed. We probably want to create a new API like > MPI_Comm_validate_many() to host these new semantics. > > It is important to remember that the validate operation changes the > communicator (primarily just the 'are_collectives_enabled' flag on the > communicator), and not anything to do with elements of the group that > form it. > > Currently after creation, a communicator does not need to track from > which communicators it was created (at least that's the way I > understand it). So creating a super communicator and calling > MPI_Comm_validate_many() on that would require such tracking to have > the validation propagate to all of the communicators that built it. So > we could do it, but the additional state tracking would force > additional memory consumption even if the operation is never used, > which is slightly problematic. > > >> >> I liked your Option B as such, however, as you point out, it has significant problems in case of applications consisting of several layers of libraries. > > > Thinking through your question above, I think Option B would require > that we track the heritage of communicators after creation, which > would increase memory consumption. It would also require us to > maintain that linkage across communicator destruction. For example, > ----------- > MPI_Comm_dup(MPI_COMM_WORLD, commA); > // MCW is linked to commA > MPI_Comm_dup(commA, commB); > // MCW is linked to commA > // commA is linked to commB > MPI_Comm_dup(commB, commC); > // MCW is linked to commA > // commA is linked to commB > // commB is linked to commC > MPI_Comm_free(commB); > // MCW is linked to commA > // commA is linked to commC (since commB is now gone) > ------------ > > In the discussion so far it seems that the inheritance is only one > way. Meaning that in the example above calling > MPI_Comm_validate_many() on commA would validate commC (and commB if > it is still around), but not MPI_COMM_WORLD. Is that what we are > looking for, or do we want it to be more complete? > > > The explicit linking in Option C puts the user in more control over > the overhead of tracking connections between communicators, but has > other issues. :/ > > -- Josh > > >> >> Sayantan. >> >>> >>> Option A: >>> Array of communicators >>> -------------------- >>> MPI_Comm_validate_many(comm[], num_comms, failed_grp) >>> Validate 'num_comms' communicators, and return a failed group. >>> ï¿½ - or - >>> MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) >>> Validate 'num_comms' communicators, and return a failed group for each >>> communicator. >>> ---- >>> >>> In this version of the operation the user passes in an array of >>> pointers to communicators. Since communicators are not often created >>> in a contiguous array, pointers to communications should probably be >>> used. The failed_grps is an array of failures in each of those >>> communicators. >>> >>> Some questions: >>> ï¿½* Should all processes pass in the same set of communicators at all >>> processes? >>> ï¿½* Should all communicators be duplicates or subsets of one another? >>> ï¿½* Does this operation run the risk of a circular dependency if the >>> user does not pass in the same set of communicators at all >>> participating processes? Is that something the MPI library should >>> protect the application from? >>> >>> >>> Option B: >>> Implicit inherited validation >>> -------------------- >>> MPI_Comm_validate_many(comm, failed_grp) >>> ---- >>> >>> The idea is to add an additional semantic (or maybe new API) to allows >>> the validation of a communicator to automatically validates all >>> communicators created from it (only dups and subsets of it?). >>> >>> The problem with this is that if an application calls >>> MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of >>> communicators that libraries might be using internally without >>> notification in those libraries. So this breaks the abstraction >>> barrier between the two in possibly a dangerous way. >>> >>> Some questions: >>> ï¿½* Are there some other semantics that we can add to help protect >>> libraries? (e.g., after implicit validation the first use of the >>> communicator will return a special error code indicating that the >>> communicator has been adjusted). >>> ï¿½* Are there thread safety issues involved with this? (e.g., the >>> library operates in a concurrent thread with its own duplicate of the >>> communicator. The application does not know about or control the >>> concurrent thread but calls MPI_Comm_validate on its own communicator >>> and implicitly changes the semantics of the duplicate communicator.) >>> ï¿½* It is only through the call to MPI_Comm_validate that we can >>> provide a uniform group of failed processes globally known. For those >>> that were implicitly validated, do we need to provide a way to access >>> this group after the call? Does this have implications on the amount >>> of storage required for this semantic? >>> >>> >>> Option C: >>> Explicit inherited validation >>> -------------------- >>> MPI_Comm_validate_link(commA, commB); >>> MPI_Comm_validate_many(commA, failed_grp) >>> /* Implies MPI_Comm_validate(commB, NULL) */ >>> >>> MPI_Comm_validate(commA, failed_grp) >>> /* Does not imply MPI_Comm_validate(commB, NULL) */ >>> ---- >>> >>> In this version the application explicitly links communicators. This >>> prevents an application from implicitly altering derived communicators >>> out of their scope (e.g., in use by other libraries). >>> >>> Some questions: >>> ï¿½* It is only through the call to MPI_Comm_validate that we can >>> provide a uniform group of failed processes globally known. For those >>> that were implicitly validated, do we need to provide a way to access >>> this group after the call (e.g., for commB)? Does this have >>> implications on the amount of storage required for this semantic? >>> ï¿½* Do we need a mechanism to 'unlink' communicators? Or determine >>> which communicators are linked? >>> ï¿½* Can a communicator be linked to multiple other communicators? >>> ï¿½* Is the linking a unidirectional operation? (so in the example above >>> validating commB does not validate commA unless there is a separate >>> MPI_Comm_validate_link(commB, commA) call) >>> >>> >>> Option D: >>> Other >>> -------------------- >>> Something else... >>> >>> >>> Thoughts? >>> >>> -- Josh >>> >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Multiple Communicator Version of MPI_Comm_validate
by Josh Hursey 15 Sep '11

15 Sep '11

On Thu, Sep 15, 2011 at 12:43 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: > > >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >> Sent: Thursday, September 15, 2011 6:13 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: Re: [Mpi3-ft] Multiple Communicator Version of >> MPI_Comm_validate >> >> On Wed, Sep 14, 2011 at 4:17 PM, Sur, Sayantan <sayantan.sur(a)intel.com> >> wrote: >> > Hi Josh, >> > >> >> >> >> Workaround: >> >> -------------------- >> >> Call MPI_Comm_validate over all of the communicators individually. >> >> This would involve 'num_comms' collective operations and likely >> impede >> >> scalability. >> >> >> >> for(i=0; i < num_comms; ++i) { >> >> ï¿½ MPI_Comm_validate(comm[i], failed_grps[i]); >> >> } >> >> >> > >> > Would it not be possible for the app to create a communicator that is >> the union of all the processes, and subsequently call validate only on >> that 'super' communicator? I hope I am not missing something from your >> example. >> >> In the current spec, no. MPI_Comm_validate only changes the state of >> the communicator passed. We probably want to create a new API like >> MPI_Comm_validate_many() to host these new semantics. > > > I agree with your comments. I agree that calling 'validate' on communicators somehow not owned (through heritage, etc.) is tricky business as it changes the semantics. > > I guess my question is that in your example comm[0..num_comms-1] are 'owned' by the layer of application code that is calling validate? There are two cases here: Yes, in that example I am assuming that the application or library layer calling the loop 'owns' all of the communicators being validated in the comm[] array. > > i) how to optimize validation for a set of possibly overlapping communicators (all owned by one layer of application code [library]) > ii) how to optimize validation for a set of communicators (possibly heritage linked by dup) across layers of libraries ... thus eliminating the requirement of each library to validate its own communicator > > Which of these cases does MPI_Comm_validate_many() aim to optimize? I think we are looking at (i). We need to be able to protect the libraries from actions outside of their communication context to the greatest degree possible. So I do not think that the application should be able to validate the communicators of the library (unless the library explicitly allows this), and visa versa. Sorry I misunderstood the first time around. -- Josh > > > Thanks. > >> >> It is important to remember that the validate operation changes the >> communicator (primarily just the 'are_collectives_enabled' flag on the >> communicator), and not anything to do with elements of the group that >> form it. >> >> Currently after creation, a communicator does not need to track from >> which communicators it was created (at least that's the way I >> understand it). So creating a super communicator and calling >> MPI_Comm_validate_many() on that would require such tracking to have >> the validation propagate to all of the communicators that built it. So >> we could do it, but the additional state tracking would force >> additional memory consumption even if the operation is never used, >> which is slightly problematic. >> >> >> > >> > I liked your Option B as such, however, as you point out, it has >> significant problems in case of applications consisting of several >> layers of libraries. >> >> >> Thinking through your question above, I think Option B would require >> that we track the heritage of communicators after creation, which >> would increase memory consumption. It would also require us to >> maintain that linkage across communicator destruction. For example, >> ----------- >> MPI_Comm_dup(MPI_COMM_WORLD, commA); >> // MCW ï¿½ is linked to commA >> MPI_Comm_dup(commA, commB); >> // MCW ï¿½ is linked to commA >> // commA is linked to commB >> MPI_Comm_dup(commB, commC); >> // MCW ï¿½ is linked to commA >> // commA is linked to commB >> // commB is linked to commC >> MPI_Comm_free(commB); >> // MCW ï¿½ is linked to commA >> // commA is linked to commC (since commB is now gone) >> ------------ >> >> In the discussion so far it seems that the inheritance is only one >> way. Meaning that in the example above calling >> MPI_Comm_validate_many() on commA would validate commC (and commB if >> it is still around), but not MPI_COMM_WORLD. Is that what we are >> looking for, or do we want it to be more complete? >> >> >> The explicit linking in Option C puts the user in more control over >> the overhead of tracking connections between communicators, but has >> other issues. :/ >> >> -- Josh >> >> >> > >> > Sayantan. >> > >> >> >> >> Option A: >> >> Array of communicators >> >> -------------------- >> >> MPI_Comm_validate_many(comm[], num_comms, failed_grp) >> >> Validate 'num_comms' communicators, and return a failed group. >> >> ï¿½ - or - >> >> MPI_Comm_validate_many(comm[], num_comms, failed_grps[]) >> >> Validate 'num_comms' communicators, and return a failed group for >> each >> >> communicator. >> >> ---- >> >> >> >> In this version of the operation the user passes in an array of >> >> pointers to communicators. Since communicators are not often created >> >> in a contiguous array, pointers to communications should probably be >> >> used. The failed_grps is an array of failures in each of those >> >> communicators. >> >> >> >> Some questions: >> >> ï¿½* Should all processes pass in the same set of communicators at all >> >> processes? >> >> ï¿½* Should all communicators be duplicates or subsets of one another? >> >> ï¿½* Does this operation run the risk of a circular dependency if the >> >> user does not pass in the same set of communicators at all >> >> participating processes? Is that something the MPI library should >> >> protect the application from? >> >> >> >> >> >> Option B: >> >> Implicit inherited validation >> >> -------------------- >> >> MPI_Comm_validate_many(comm, failed_grp) >> >> ---- >> >> >> >> The idea is to add an additional semantic (or maybe new API) to >> allows >> >> the validation of a communicator to automatically validates all >> >> communicators created from it (only dups and subsets of it?). >> >> >> >> The problem with this is that if an application calls >> >> MPI_Comm_validate on MPI_COMM_WORLD, it changes the semantics of >> >> communicators that libraries might be using internally without >> >> notification in those libraries. So this breaks the abstraction >> >> barrier between the two in possibly a dangerous way. >> >> >> >> Some questions: >> >> ï¿½* Are there some other semantics that we can add to help protect >> >> libraries? (e.g., after implicit validation the first use of the >> >> communicator will return a special error code indicating that the >> >> communicator has been adjusted). >> >> ï¿½* Are there thread safety issues involved with this? (e.g., the >> >> library operates in a concurrent thread with its own duplicate of >> the >> >> communicator. The application does not know about or control the >> >> concurrent thread but calls MPI_Comm_validate on its own >> communicator >> >> and implicitly changes the semantics of the duplicate communicator.) >> >> ï¿½* It is only through the call to MPI_Comm_validate that we can >> >> provide a uniform group of failed processes globally known. For >> those >> >> that were implicitly validated, do we need to provide a way to >> access >> >> this group after the call? Does this have implications on the amount >> >> of storage required for this semantic? >> >> >> >> >> >> Option C: >> >> Explicit inherited validation >> >> -------------------- >> >> MPI_Comm_validate_link(commA, commB); >> >> MPI_Comm_validate_many(commA, failed_grp) >> >> /* Implies MPI_Comm_validate(commB, NULL) */ >> >> >> >> MPI_Comm_validate(commA, failed_grp) >> >> /* Does not imply MPI_Comm_validate(commB, NULL) */ >> >> ---- >> >> >> >> In this version the application explicitly links communicators. This >> >> prevents an application from implicitly altering derived >> communicators >> >> out of their scope (e.g., in use by other libraries). >> >> >> >> Some questions: >> >> ï¿½* It is only through the call to MPI_Comm_validate that we can >> >> provide a uniform group of failed processes globally known. For >> those >> >> that were implicitly validated, do we need to provide a way to >> access >> >> this group after the call (e.g., for commB)? Does this have >> >> implications on the amount of storage required for this semantic? >> >> ï¿½* Do we need a mechanism to 'unlink' communicators? Or determine >> >> which communicators are linked? >> >> ï¿½* Can a communicator be linked to multiple other communicators? >> >> ï¿½* Is the linking a unidirectional operation? (so in the example >> above >> >> validating commB does not validate commA unless there is a separate >> >> MPI_Comm_validate_link(commB, commA) call) >> >> >> >> >> >> Option D: >> >> Other >> >> -------------------- >> >> Something else... >> >> >> >> >> >> Thoughts? >> >> >> >> -- Josh >> >> >> >> >> >> -- >> >> Joshua Hursey >> >> Postdoctoral Research Associate >> >> Oak Ridge National Laboratory >> >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > >> > _______________________________________________ >> > mpi3-ft mailing list >> > mpi3-ft(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > >> > >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0