December 2013 - mpiwg-ft - mailman3.cels.anl.gov

2013-12-10 – MPI Forum Discussion
by Wesley Bland 10 Dec '13

10 Dec '13

The notes for the combined meetings between yesterday and today have now been posted to the wiki. Take a look and let me know if I’ve forgotten anything. https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-12-10 Thanks, Wesley

1 0

2013-12-10 FTWG Discussion
by Wesley Bland 10 Dec '13

10 Dec '13

Here’s the notes from the discussions from the FTWG today. Thanks, Wesley

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by Jim Dinan 09 Dec '13

09 Dec '13

Assuming no more travel glitches, I should make it for our 4pm WG meeting. *cross-fingers* On Dec 9, 2013 6:59 AM, "Jim Dinan" <james.dinan(a)gmail.com> wrote: > It would be helpful to me if we could have a longer discussion about this. > If we have to make changes to matching, that will have significant impact, > especially on offload networks (e.g. limited to 64 bits of matching). I > would like to better understand how this works when the incarnation is not > a part of the match, and how you can ensure that messages from a previous > incarnation don't match on a reused context ID. > > Unfortunately, United canceled my flight from BOS->ORD this morning, and > I'm not sure I'll make it for the WG. It would be great if this topic can > still be discussed in the WR. We should make sure we have a complete story > around how the proposal impacts matching and how an implementation that > does not include incarnation number in the match functions reliably. > > ~Jim. > > > On Mon, Dec 9, 2013 at 12:01 AM, George Bosilca <bosilca(a)icl.utk.edu>wrote: > >> Jim, >> The incarnation number is just a way, a mere optimization not needed for >> correctness purposes. In fact, any MPI library currently supporting >> non-collective MPI_COM_FREE face a similar issue today. >> >> Regarding the usage of an incarnation number, you are right, there is one >> per communicator, it is part of the matching and must be monotonic. Its >> size should be large enough to prevent the creation of coms with the same >> id, while keeping in mind that creating a communicator is a synchronous >> operation among participants. Thus, it can roll over without any issues in >> case you reach three upper bound. >> >> George. >> On Dec 9, 2013 2:14 AM, "Jim Dinan" <james.dinan(a)gmail.com> wrote: >> >>> Hi Guys, >>> >>> Thanks for the detailed responses (and sorry for going off-topic). >>> >>> Does the incarnation number have to be included in matching? >>> >>> I assume that we would need to track an incarnation number for each >>> context ID. When creating a communicator, we can't guarantee that you'll >>> get the same incarnation number everywhere for a given Context ID. Do you >>> select the highest incarnation number from among the processes involved? I >>> assume the incarnation counter is meant to be monotonic increasing -- what >>> happens when it reaches its maximum value? >>> >>> ~Jim. >>> >>> >>> On Thu, Dec 5, 2013 at 1:55 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >>> >>>> Yes you can free the revoked communicator so you can also reuse the >>>> context id. There are a few ways you can handle this. One is to add an >>>> incarnation number to the communicator or process (which is the route >>>> chosen by the UTK ULFM implementation). Another is that once you know a >>>> process has failed, you ignore all future messages from that process (this >>>> causes issues if you want to deal with transient failures, but we’ve >>>> defined the scope of the work to exclude those). I’m sure there are more, >>>> possibly cleverer ways of solving this but those came to mind first. >>>> >>>> Wesley >>>> >>>> >>>> On Dec 5, 2013, at 12:39 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>> >>>> This is a little off-topic from the current discussion -- can revoking >>>> communicators could lead to leaking context IDs? After revoking a >>>> communicator, can you safely free it and then reuse the same context ID >>>> without concerns about erroneous messages arriving with that context ID? >>>> Or is that context ID dead for the remainder of the execution? >>>> >>>> ~Jim. >>>> >>>> >>>> On Thu, Dec 5, 2013 at 10:49 AM, Wesley Bland <wbland(a)mcs.anl.gov>wrote: >>>> >>>>> I think there’s still confusion here between the current FT proposal >>>>> and the roadmap for previous proposals. I think the previous proposal was >>>>> designed to be a stopgap solution that would allow MPI to stabilize, but >>>>> not necessarily fully recover without a lot of work. The current proposal >>>>> provides all the tools necessary for both stabilization and recovery. Thus >>>>> we haven’t been discussing the “next step”, because I think most of us see >>>>> the next step not as a standardization effort for failure recovery, but >>>>> something outside of the standard such as the libraries that will make FT >>>>> easier to use. >>>>> >>>>> It’s true that MPI_COMM_REVOKE will make a current communicator >>>>> unusable for further communication. Period. However, you can still query >>>>> all of the local data about the communicator (rank, size, topology, info >>>>> keys, etc.) to allow you to reconstruct a new communicator. That >>>>> reconstruction is very much possible with the current proposal. It’s >>>>> obviously not cheap, but I think we all know that FT recovery isn’t >>>>> necessarily cheap. I think somewhere along the way we even had an example >>>>> of how you could reconstruct a communicator with processes retaining their >>>>> original ranks (though if you’re not going to replace the failed processes, >>>>> I’m not sure what use it is to retain your rank since your algorithm will >>>>> need to be able to handle such things anyway). >>>>> >>>>> MPI_COMM_REVOKE is the mechanism by which a process notifies other >>>>> ranks of errors. No user-level protocol is necessary, though there are >>>>> specific cases where a user-level solution might be a better solution >>>>> depending on the communication patter. When you call MPI_COMM_REVOKE, you >>>>> give up on any remaining (incoming) traffic that’s outstanding on the >>>>> communicator. For remote processes, they too will not receive any messages >>>>> after they return the error code MPI_COMM_REVOKED. This doesn’t preclude >>>>> them from delaying reporting MPI_ERR_REVOKED while they flush any messages >>>>> that have already been received, however, once the error code is returned, >>>>> the communicator is dead. Users have to be able to reason about this and I >>>>> don’t think it’s entirely unclear. Once revoke is called, that communicator >>>>> is hosed. You can create a new communicator and use it to decide what’s >>>>> necessary to recover your application (algorithm state, data repair, etc.). >>>>> That’s the guarantee that users get: once the communicator is revoked, all >>>>> messages/requests that have not been received are cancelled and will never >>>>> be completed. Any other guarantee invites lots of trouble as you have to >>>>> deal with race conditions involving who receives the revoke notification >>>>> before any other messages and whether or not all of the links necessary to >>>>> complete a message transfer are still available. The entire point of the >>>>> REVOKE call is a last resort operation to prevent deadlock. It’s entirely >>>>> possible that an application won’t need to use the revoke if there aren’t >>>>> complex communication dependencies. It may be possible to take care of >>>>> things at the user level in a way that’s cheaper than using REVOKE. >>>>> >>>>> The entire basis of this proposal is that we can’t provide a fault >>>>> tolerance solution that will cheaply stabilize MPI, repair communicators, >>>>> and notify all processes of failures. It’s possible to do all of these >>>>> things in a way that’s not outrageously expensive, but they require >>>>> cooperation from the application's perspective (or libraries that act on >>>>> the application’s behalf). It’s possible to do many of those things, but >>>>> they are so costly that they would never be approved. It’s much more >>>>> effective to do them on top of MPI. We’re just providing the low level >>>>> tools necessary to make it happen. >>>>> >>>>> Wesley >>>>> >>>>> On Dec 5, 2013, at 8:14 AM, Richard Graham <richardg(a)mellanox.com> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> *From:* mpiwg-ft [mailto:[email protected]<mpiwg-ft-bounces(a)lists.mpi-forum.org> >>>>> ] *On Behalf Of *George Bosilca >>>>> *Sent:* Wednesday, November 27, 2013 3:35 PM >>>>> *To:* MPI WG Fault Tolerance and Dynamic Process Control working Group >>>>> *Subject:* Re: [mpiwg-ft] MPI_Comm_revoke behavior >>>>> >>>>> >>>>> >>>>> On Nov 27, 2013, at 20:54 , Richard Graham <richardg(a)mellanox.com> >>>>> wrote: >>>>> >>>>> >>>>> On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> I am thinking about the next step, and have some questions on the >>>>> semantics of MPI_Comm_revoke() >>>>> >>>>> What next step are you referring to? >>>>> >>>>> [rich] To the full recovery stage. Post what we are talking about now. >>>>> >>>>> >>>>> Full recovery stage? Can you expose a little more details here please. >>>>> >>>>> [rich] the original intent was to allow for full restoration of >>>>> communicators after failure, with minimal impact on those ranks that did >>>>> not fail (don’t want to get into what that means now …). Those goals were >>>>> reduced for pragmatic reasons. I want to make sure that when/if there is >>>>> work continued in this direction, the current proposal does not preclude >>>>> this. One of the issues raised to me recently is that after a revoke one >>>>> will not be able to accomplish such a goal on the remaining ranks – e.g., >>>>> ranks will be reassigned. I am following up very specifically on this >>>>> question. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> - When the routine returns, can the communicator ever be >>>>> used again ? If I remember correctly, the communicator is available for >>>>> point-to-point traffic, but not collective traffic – is this correct ? >>>>> >>>>> A revoked communicator is unable to support any communication >>>>> (point-to-point or collective) with the exception of agree and shrink. If >>>>> this is not clear enough in the current version of the proposal we should >>>>> definitively address it. >>>>> >>>>> [rich] does this mean all current state (aside from who is alive) >>>>> associated with the communicator is gone ? >>>>> >>>>> >>>>> Every deterministic information is still available (info and >>>>> attributes). You can look for the group of processes associated with the >>>>> communicator, as well as the group of failed. If what you are looking for >>>>> is the possible unexpected messages, this is up to the implementation (see >>>>> below). >>>>> [rich] don’t understand >>>>> >>>>> >>>>> Can’t rely on continuing sending pending messages ? >>>>> >>>>> >>>>> Not on a revoked communicator. If continuing to exchange messages is a >>>>> requirement, the communicator should not be revoked. >>>>> [rich] How does one then notify other ranks of the errors – does this >>>>> have to be a user-level protocol ? >>>>> >>>>> >>>>> >>>>> Looking forward, if one wants to restart the failed ranks >>>>> (let’s assume we add support for this), what can be assume about the >>>>> “repaired” communicator ? What can’t I assume about this communicator ? >>>>> >>>>> What you can assume depends on what is the meaning of “repaired”. >>>>> Already today one can spawn new processes and reconstruct a communicator >>>>> identical to the original communicator before any fault. This can be done >>>>> using MPI dynamics together with the agreement available in the ULFM >>>>> proposal. >>>>> >>>>> [rich] This implies that all outstanding traffic is flushed – is this >>>>> correct ? >>>>> >>>>> >>>>> This is up to the MPI implementation. This is specified on the first >>>>> “Advice to implementors” on the second page. >>>>> [rich] does not seem like a good idea – users should have guarantees >>>>> on what they get if they use MPI. >>>>> >>>>> George. >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpiwg-ft mailing list >>>>> mpiwg-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpiwg-ft mailing list >>>>> mpiwg-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > >

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by Jim Dinan 09 Dec '13

09 Dec '13

It would be helpful to me if we could have a longer discussion about this. If we have to make changes to matching, that will have significant impact, especially on offload networks (e.g. limited to 64 bits of matching). I would like to better understand how this works when the incarnation is not a part of the match, and how you can ensure that messages from a previous incarnation don't match on a reused context ID. Unfortunately, United canceled my flight from BOS->ORD this morning, and I'm not sure I'll make it for the WG. It would be great if this topic can still be discussed in the WR. We should make sure we have a complete story around how the proposal impacts matching and how an implementation that does not include incarnation number in the match functions reliably. ~Jim. On Mon, Dec 9, 2013 at 12:01 AM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > Jim, > The incarnation number is just a way, a mere optimization not needed for > correctness purposes. In fact, any MPI library currently supporting > non-collective MPI_COM_FREE face a similar issue today. > > Regarding the usage of an incarnation number, you are right, there is one > per communicator, it is part of the matching and must be monotonic. Its > size should be large enough to prevent the creation of coms with the same > id, while keeping in mind that creating a communicator is a synchronous > operation among participants. Thus, it can roll over without any issues in > case you reach three upper bound. > > George. > On Dec 9, 2013 2:14 AM, "Jim Dinan" <james.dinan(a)gmail.com> wrote: > >> Hi Guys, >> >> Thanks for the detailed responses (and sorry for going off-topic). >> >> Does the incarnation number have to be included in matching? >> >> I assume that we would need to track an incarnation number for each >> context ID. When creating a communicator, we can't guarantee that you'll >> get the same incarnation number everywhere for a given Context ID. Do you >> select the highest incarnation number from among the processes involved? I >> assume the incarnation counter is meant to be monotonic increasing -- what >> happens when it reaches its maximum value? >> >> ~Jim. >> >> >> On Thu, Dec 5, 2013 at 1:55 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> >>> Yes you can free the revoked communicator so you can also reuse the >>> context id. There are a few ways you can handle this. One is to add an >>> incarnation number to the communicator or process (which is the route >>> chosen by the UTK ULFM implementation). Another is that once you know a >>> process has failed, you ignore all future messages from that process (this >>> causes issues if you want to deal with transient failures, but we’ve >>> defined the scope of the work to exclude those). I’m sure there are more, >>> possibly cleverer ways of solving this but those came to mind first. >>> >>> Wesley >>> >>> >>> On Dec 5, 2013, at 12:39 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>> >>> This is a little off-topic from the current discussion -- can revoking >>> communicators could lead to leaking context IDs? After revoking a >>> communicator, can you safely free it and then reuse the same context ID >>> without concerns about erroneous messages arriving with that context ID? >>> Or is that context ID dead for the remainder of the execution? >>> >>> ~Jim. >>> >>> >>> On Thu, Dec 5, 2013 at 10:49 AM, Wesley Bland <wbland(a)mcs.anl.gov>wrote: >>> >>>> I think there’s still confusion here between the current FT proposal >>>> and the roadmap for previous proposals. I think the previous proposal was >>>> designed to be a stopgap solution that would allow MPI to stabilize, but >>>> not necessarily fully recover without a lot of work. The current proposal >>>> provides all the tools necessary for both stabilization and recovery. Thus >>>> we haven’t been discussing the “next step”, because I think most of us see >>>> the next step not as a standardization effort for failure recovery, but >>>> something outside of the standard such as the libraries that will make FT >>>> easier to use. >>>> >>>> It’s true that MPI_COMM_REVOKE will make a current communicator >>>> unusable for further communication. Period. However, you can still query >>>> all of the local data about the communicator (rank, size, topology, info >>>> keys, etc.) to allow you to reconstruct a new communicator. That >>>> reconstruction is very much possible with the current proposal. It’s >>>> obviously not cheap, but I think we all know that FT recovery isn’t >>>> necessarily cheap. I think somewhere along the way we even had an example >>>> of how you could reconstruct a communicator with processes retaining their >>>> original ranks (though if you’re not going to replace the failed processes, >>>> I’m not sure what use it is to retain your rank since your algorithm will >>>> need to be able to handle such things anyway). >>>> >>>> MPI_COMM_REVOKE is the mechanism by which a process notifies other >>>> ranks of errors. No user-level protocol is necessary, though there are >>>> specific cases where a user-level solution might be a better solution >>>> depending on the communication patter. When you call MPI_COMM_REVOKE, you >>>> give up on any remaining (incoming) traffic that’s outstanding on the >>>> communicator. For remote processes, they too will not receive any messages >>>> after they return the error code MPI_COMM_REVOKED. This doesn’t preclude >>>> them from delaying reporting MPI_ERR_REVOKED while they flush any messages >>>> that have already been received, however, once the error code is returned, >>>> the communicator is dead. Users have to be able to reason about this and I >>>> don’t think it’s entirely unclear. Once revoke is called, that communicator >>>> is hosed. You can create a new communicator and use it to decide what’s >>>> necessary to recover your application (algorithm state, data repair, etc.). >>>> That’s the guarantee that users get: once the communicator is revoked, all >>>> messages/requests that have not been received are cancelled and will never >>>> be completed. Any other guarantee invites lots of trouble as you have to >>>> deal with race conditions involving who receives the revoke notification >>>> before any other messages and whether or not all of the links necessary to >>>> complete a message transfer are still available. The entire point of the >>>> REVOKE call is a last resort operation to prevent deadlock. It’s entirely >>>> possible that an application won’t need to use the revoke if there aren’t >>>> complex communication dependencies. It may be possible to take care of >>>> things at the user level in a way that’s cheaper than using REVOKE. >>>> >>>> The entire basis of this proposal is that we can’t provide a fault >>>> tolerance solution that will cheaply stabilize MPI, repair communicators, >>>> and notify all processes of failures. It’s possible to do all of these >>>> things in a way that’s not outrageously expensive, but they require >>>> cooperation from the application's perspective (or libraries that act on >>>> the application’s behalf). It’s possible to do many of those things, but >>>> they are so costly that they would never be approved. It’s much more >>>> effective to do them on top of MPI. We’re just providing the low level >>>> tools necessary to make it happen. >>>> >>>> Wesley >>>> >>>> On Dec 5, 2013, at 8:14 AM, Richard Graham <richardg(a)mellanox.com> >>>> wrote: >>>> >>>> >>>> >>>> *From:* mpiwg-ft [mailto:[email protected]<mpiwg-ft-bounces(a)lists.mpi-forum.org> >>>> ] *On Behalf Of *George Bosilca >>>> *Sent:* Wednesday, November 27, 2013 3:35 PM >>>> *To:* MPI WG Fault Tolerance and Dynamic Process Control working Group >>>> *Subject:* Re: [mpiwg-ft] MPI_Comm_revoke behavior >>>> >>>> >>>> >>>> On Nov 27, 2013, at 20:54 , Richard Graham <richardg(a)mellanox.com> >>>> wrote: >>>> >>>> >>>> On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> >>>> wrote: >>>> >>>> >>>> >>>> I am thinking about the next step, and have some questions on the >>>> semantics of MPI_Comm_revoke() >>>> >>>> What next step are you referring to? >>>> >>>> [rich] To the full recovery stage. Post what we are talking about now. >>>> >>>> >>>> Full recovery stage? Can you expose a little more details here please. >>>> >>>> [rich] the original intent was to allow for full restoration of >>>> communicators after failure, with minimal impact on those ranks that did >>>> not fail (don’t want to get into what that means now …). Those goals were >>>> reduced for pragmatic reasons. I want to make sure that when/if there is >>>> work continued in this direction, the current proposal does not preclude >>>> this. One of the issues raised to me recently is that after a revoke one >>>> will not be able to accomplish such a goal on the remaining ranks – e.g., >>>> ranks will be reassigned. I am following up very specifically on this >>>> question. >>>> >>>> >>>> >>>> >>>> >>>> - When the routine returns, can the communicator ever be >>>> used again ? If I remember correctly, the communicator is available for >>>> point-to-point traffic, but not collective traffic – is this correct ? >>>> >>>> A revoked communicator is unable to support any communication >>>> (point-to-point or collective) with the exception of agree and shrink. If >>>> this is not clear enough in the current version of the proposal we should >>>> definitively address it. >>>> >>>> [rich] does this mean all current state (aside from who is alive) >>>> associated with the communicator is gone ? >>>> >>>> >>>> Every deterministic information is still available (info and >>>> attributes). You can look for the group of processes associated with the >>>> communicator, as well as the group of failed. If what you are looking for >>>> is the possible unexpected messages, this is up to the implementation (see >>>> below). >>>> [rich] don’t understand >>>> >>>> >>>> Can’t rely on continuing sending pending messages ? >>>> >>>> >>>> Not on a revoked communicator. If continuing to exchange messages is a >>>> requirement, the communicator should not be revoked. >>>> [rich] How does one then notify other ranks of the errors – does this >>>> have to be a user-level protocol ? >>>> >>>> >>>> >>>> Looking forward, if one wants to restart the failed ranks >>>> (let’s assume we add support for this), what can be assume about the >>>> “repaired” communicator ? What can’t I assume about this communicator ? >>>> >>>> What you can assume depends on what is the meaning of “repaired”. >>>> Already today one can spawn new processes and reconstruct a communicator >>>> identical to the original communicator before any fault. This can be done >>>> using MPI dynamics together with the agreement available in the ULFM >>>> proposal. >>>> >>>> [rich] This implies that all outstanding traffic is flushed – is this >>>> correct ? >>>> >>>> >>>> This is up to the MPI implementation. This is specified on the first >>>> “Advice to implementors” on the second page. >>>> [rich] does not seem like a good idea – users should have guarantees >>>> on what they get if they use MPI. >>>> >>>> George. >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by George Bosilca 08 Dec '13

08 Dec '13

Jim, The incarnation number is just a way, a mere optimization not needed for correctness purposes. In fact, any MPI library currently supporting non-collective MPI_COM_FREE face a similar issue today. Regarding the usage of an incarnation number, you are right, there is one per communicator, it is part of the matching and must be monotonic. Its size should be large enough to prevent the creation of coms with the same id, while keeping in mind that creating a communicator is a synchronous operation among participants. Thus, it can roll over without any issues in case you reach three upper bound. George. On Dec 9, 2013 2:14 AM, "Jim Dinan" <james.dinan(a)gmail.com> wrote: > Hi Guys, > > Thanks for the detailed responses (and sorry for going off-topic). > > Does the incarnation number have to be included in matching? > > I assume that we would need to track an incarnation number for each > context ID. When creating a communicator, we can't guarantee that you'll > get the same incarnation number everywhere for a given Context ID. Do you > select the highest incarnation number from among the processes involved? I > assume the incarnation counter is meant to be monotonic increasing -- what > happens when it reaches its maximum value? > > ~Jim. > > > On Thu, Dec 5, 2013 at 1:55 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> Yes you can free the revoked communicator so you can also reuse the >> context id. There are a few ways you can handle this. One is to add an >> incarnation number to the communicator or process (which is the route >> chosen by the UTK ULFM implementation). Another is that once you know a >> process has failed, you ignore all future messages from that process (this >> causes issues if you want to deal with transient failures, but we’ve >> defined the scope of the work to exclude those). I’m sure there are more, >> possibly cleverer ways of solving this but those came to mind first. >> >> Wesley >> >> >> On Dec 5, 2013, at 12:39 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >> This is a little off-topic from the current discussion -- can revoking >> communicators could lead to leaking context IDs? After revoking a >> communicator, can you safely free it and then reuse the same context ID >> without concerns about erroneous messages arriving with that context ID? >> Or is that context ID dead for the remainder of the execution? >> >> ~Jim. >> >> >> On Thu, Dec 5, 2013 at 10:49 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> >>> I think there’s still confusion here between the current FT proposal and >>> the roadmap for previous proposals. I think the previous proposal was >>> designed to be a stopgap solution that would allow MPI to stabilize, but >>> not necessarily fully recover without a lot of work. The current proposal >>> provides all the tools necessary for both stabilization and recovery. Thus >>> we haven’t been discussing the “next step”, because I think most of us see >>> the next step not as a standardization effort for failure recovery, but >>> something outside of the standard such as the libraries that will make FT >>> easier to use. >>> >>> It’s true that MPI_COMM_REVOKE will make a current communicator unusable >>> for further communication. Period. However, you can still query all of the >>> local data about the communicator (rank, size, topology, info keys, etc.) >>> to allow you to reconstruct a new communicator. That reconstruction is very >>> much possible with the current proposal. It’s obviously not cheap, but I >>> think we all know that FT recovery isn’t necessarily cheap. I think >>> somewhere along the way we even had an example of how you could reconstruct >>> a communicator with processes retaining their original ranks (though if >>> you’re not going to replace the failed processes, I’m not sure what use it >>> is to retain your rank since your algorithm will need to be able to handle >>> such things anyway). >>> >>> MPI_COMM_REVOKE is the mechanism by which a process notifies other ranks >>> of errors. No user-level protocol is necessary, though there are specific >>> cases where a user-level solution might be a better solution depending on >>> the communication patter. When you call MPI_COMM_REVOKE, you give up on any >>> remaining (incoming) traffic that’s outstanding on the communicator. For >>> remote processes, they too will not receive any messages after they return >>> the error code MPI_COMM_REVOKED. This doesn’t preclude them from delaying >>> reporting MPI_ERR_REVOKED while they flush any messages that have already >>> been received, however, once the error code is returned, the communicator >>> is dead. Users have to be able to reason about this and I don’t think it’s >>> entirely unclear. Once revoke is called, that communicator is hosed. You >>> can create a new communicator and use it to decide what’s necessary to >>> recover your application (algorithm state, data repair, etc.). That’s the >>> guarantee that users get: once the communicator is revoked, all >>> messages/requests that have not been received are cancelled and will never >>> be completed. Any other guarantee invites lots of trouble as you have to >>> deal with race conditions involving who receives the revoke notification >>> before any other messages and whether or not all of the links necessary to >>> complete a message transfer are still available. The entire point of the >>> REVOKE call is a last resort operation to prevent deadlock. It’s entirely >>> possible that an application won’t need to use the revoke if there aren’t >>> complex communication dependencies. It may be possible to take care of >>> things at the user level in a way that’s cheaper than using REVOKE. >>> >>> The entire basis of this proposal is that we can’t provide a fault >>> tolerance solution that will cheaply stabilize MPI, repair communicators, >>> and notify all processes of failures. It’s possible to do all of these >>> things in a way that’s not outrageously expensive, but they require >>> cooperation from the application's perspective (or libraries that act on >>> the application’s behalf). It’s possible to do many of those things, but >>> they are so costly that they would never be approved. It’s much more >>> effective to do them on top of MPI. We’re just providing the low level >>> tools necessary to make it happen. >>> >>> Wesley >>> >>> On Dec 5, 2013, at 8:14 AM, Richard Graham <richardg(a)mellanox.com> >>> wrote: >>> >>> >>> >>> *From:* mpiwg-ft [mailto:[email protected]<mpiwg-ft-bounces(a)lists.mpi-forum.org> >>> ] *On Behalf Of *George Bosilca >>> *Sent:* Wednesday, November 27, 2013 3:35 PM >>> *To:* MPI WG Fault Tolerance and Dynamic Process Control working Group >>> *Subject:* Re: [mpiwg-ft] MPI_Comm_revoke behavior >>> >>> >>> >>> On Nov 27, 2013, at 20:54 , Richard Graham <richardg(a)mellanox.com> >>> wrote: >>> >>> >>> On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> >>> wrote: >>> >>> >>> >>> I am thinking about the next step, and have some questions on the >>> semantics of MPI_Comm_revoke() >>> >>> What next step are you referring to? >>> >>> [rich] To the full recovery stage. Post what we are talking about now. >>> >>> >>> Full recovery stage? Can you expose a little more details here please. >>> >>> [rich] the original intent was to allow for full restoration of >>> communicators after failure, with minimal impact on those ranks that did >>> not fail (don’t want to get into what that means now …). Those goals were >>> reduced for pragmatic reasons. I want to make sure that when/if there is >>> work continued in this direction, the current proposal does not preclude >>> this. One of the issues raised to me recently is that after a revoke one >>> will not be able to accomplish such a goal on the remaining ranks – e.g., >>> ranks will be reassigned. I am following up very specifically on this >>> question. >>> >>> >>> >>> >>> >>> - When the routine returns, can the communicator ever be used >>> again ? If I remember correctly, the communicator is available for >>> point-to-point traffic, but not collective traffic – is this correct ? >>> >>> A revoked communicator is unable to support any communication >>> (point-to-point or collective) with the exception of agree and shrink. If >>> this is not clear enough in the current version of the proposal we should >>> definitively address it. >>> >>> [rich] does this mean all current state (aside from who is alive) >>> associated with the communicator is gone ? >>> >>> >>> Every deterministic information is still available (info and >>> attributes). You can look for the group of processes associated with the >>> communicator, as well as the group of failed. If what you are looking for >>> is the possible unexpected messages, this is up to the implementation (see >>> below). >>> [rich] don’t understand >>> >>> >>> Can’t rely on continuing sending pending messages ? >>> >>> >>> Not on a revoked communicator. If continuing to exchange messages is a >>> requirement, the communicator should not be revoked. >>> [rich] How does one then notify other ranks of the errors – does this >>> have to be a user-level protocol ? >>> >>> >>> >>> Looking forward, if one wants to restart the failed ranks >>> (let’s assume we add support for this), what can be assume about the >>> “repaired” communicator ? What can’t I assume about this communicator ? >>> >>> What you can assume depends on what is the meaning of “repaired”. >>> Already today one can spawn new processes and reconstruct a communicator >>> identical to the original communicator before any fault. This can be done >>> using MPI dynamics together with the agreement available in the ULFM >>> proposal. >>> >>> [rich] This implies that all outstanding traffic is flushed – is this >>> correct ? >>> >>> >>> This is up to the MPI implementation. This is specified on the first >>> “Advice to implementors” on the second page. >>> [rich] does not seem like a good idea – users should have guarantees on >>> what they get if they use MPI. >>> >>> George. >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by Aurélien Bouteiller 08 Dec '13

08 Dec '13

Le 8 déc. 2013 à 20:14, Jim Dinan <james.dinan(a)gmail.com> a écrit : > Hi Guys, > > Thanks for the detailed responses (and sorry for going off-topic). > > Does the incarnation number have to be included in matching? > No it doesn’t. The only use of incarnate numbers is to protect against revoke messages that risk revoking a new communicator that happens to have the same cid. For matching etc, the normal logic of correct cid allocation works as normal. Aurelien > I assume that we would need to track an incarnation number for each context ID. When creating a communicator, we can't guarantee that you'll get the same incarnation number everywhere for a given Context ID. Do you select the highest incarnation number from among the processes involved? I assume the incarnation counter is meant to be monotonic increasing Yes. > — what happens when it reaches its maximum value? > It cycles to 0. would not expect this to be a practical issue, except if you create 4 billions comms in less than the typical cross-network latency :D Aurelien > ~Jim. > > > On Thu, Dec 5, 2013 at 1:55 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > Yes you can free the revoked communicator so you can also reuse the context id. There are a few ways you can handle this. One is to add an incarnation number to the communicator or process (which is the route chosen by the UTK ULFM implementation). Another is that once you know a process has failed, you ignore all future messages from that process (this causes issues if you want to deal with transient failures, but we’ve defined the scope of the work to exclude those). I’m sure there are more, possibly cleverer ways of solving this but those came to mind first. > > Wesley > > > On Dec 5, 2013, at 12:39 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> This is a little off-topic from the current discussion -- can revoking communicators could lead to leaking context IDs? After revoking a communicator, can you safely free it and then reuse the same context ID without concerns about erroneous messages arriving with that context ID? Or is that context ID dead for the remainder of the execution? >> >> ~Jim. >> >> >> On Thu, Dec 5, 2013 at 10:49 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> I think there’s still confusion here between the current FT proposal and the roadmap for previous proposals. I think the previous proposal was designed to be a stopgap solution that would allow MPI to stabilize, but not necessarily fully recover without a lot of work. The current proposal provides all the tools necessary for both stabilization and recovery. Thus we haven’t been discussing the “next step”, because I think most of us see the next step not as a standardization effort for failure recovery, but something outside of the standard such as the libraries that will make FT easier to use. >> >> It’s true that MPI_COMM_REVOKE will make a current communicator unusable for further communication. Period. However, you can still query all of the local data about the communicator (rank, size, topology, info keys, etc.) to allow you to reconstruct a new communicator. That reconstruction is very much possible with the current proposal. It’s obviously not cheap, but I think we all know that FT recovery isn’t necessarily cheap. I think somewhere along the way we even had an example of how you could reconstruct a communicator with processes retaining their original ranks (though if you’re not going to replace the failed processes, I’m not sure what use it is to retain your rank since your algorithm will need to be able to handle such things anyway). >> >> MPI_COMM_REVOKE is the mechanism by which a process notifies other ranks of errors. No user-level protocol is necessary, though there are specific cases where a user-level solution might be a better solution depending on the communication patter. When you call MPI_COMM_REVOKE, you give up on any remaining (incoming) traffic that’s outstanding on the communicator. For remote processes, they too will not receive any messages after they return the error code MPI_COMM_REVOKED. This doesn’t preclude them from delaying reporting MPI_ERR_REVOKED while they flush any messages that have already been received, however, once the error code is returned, the communicator is dead. Users have to be able to reason about this and I don’t think it’s entirely unclear. Once revoke is called, that communicator is hosed. You can create a new communicator and use it to decide what’s necessary to recover your application (algorithm state, data repair, etc.). That’s the guarantee that users get: once the communicator is revoked, all messages/requests that have not been received are cancelled and will never be completed. Any other guarantee invites lots of trouble as you have to deal with race conditions involving who receives the revoke notification before any other messages and whether or not all of the links necessary to complete a message transfer are still available. The entire point of the REVOKE call is a last resort operation to prevent deadlock. It’s entirely possible that an application won’t need to use the revoke if there aren’t complex communication dependencies. It may be possible to take care of things at the user level in a way that’s cheaper than using REVOKE. >> >> The entire basis of this proposal is that we can’t provide a fault tolerance solution that will cheaply stabilize MPI, repair communicators, and notify all processes of failures. It’s possible to do all of these things in a way that’s not outrageously expensive, but they require cooperation from the application's perspective (or libraries that act on the application’s behalf). It’s possible to do many of those things, but they are so costly that they would never be approved. It’s much more effective to do them on top of MPI. We’re just providing the low level tools necessary to make it happen. >> >> Wesley >> >> On Dec 5, 2013, at 8:14 AM, Richard Graham <richardg(a)mellanox.com> wrote: >> >>> >>> >>> From: mpiwg-ft [mailto:[email protected]] On Behalf Of George Bosilca >>> Sent: Wednesday, November 27, 2013 3:35 PM >>> To: MPI WG Fault Tolerance and Dynamic Process Control working Group >>> Subject: Re: [mpiwg-ft] MPI_Comm_revoke behavior >>> >>> >>> On Nov 27, 2013, at 20:54 , Richard Graham <richardg(a)mellanox.com> wrote: >>> >>> >>> On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> wrote: >>> >>> >>> >>> I am thinking about the next step, and have some questions on the semantics of MPI_Comm_revoke() >>> >>> What next step are you referring to? >>> [rich] To the full recovery stage. Post what we are talking about now. >>> >>> Full recovery stage? Can you expose a little more details here please. >>> [rich] the original intent was to allow for full restoration of communicators after failure, with minimal impact on those ranks that did not fail (don’t want to get into what that means now …). Those goals were reduced for pragmatic reasons. I want to make sure that when/if there is work continued in this direction, the current proposal does not preclude this. One of the issues raised to me recently is that after a revoke one will not be able to accomplish such a goal on the remaining ranks – e.g., ranks will be reassigned. I am following up very specifically on this question. >>> >>> >>> >>> >>> - When the routine returns, can the communicator ever be used again ? If I remember correctly, the communicator is available for point-to-point traffic, but not collective traffic – is this correct ? >>> >>> A revoked communicator is unable to support any communication (point-to-point or collective) with the exception of agree and shrink. If this is not clear enough in the current version of the proposal we should definitively address it. >>> [rich] does this mean all current state (aside from who is alive) associated with the communicator is gone ? >>> >>> Every deterministic information is still available (info and attributes). You can look for the group of processes associated with the communicator, as well as the group of failed. If what you are looking for is the possible unexpected messages, this is up to the implementation (see below). >>> [rich] don’t understand >>> >>> Can’t rely on continuing sending pending messages ? >>> >>> Not on a revoked communicator. If continuing to exchange messages is a requirement, the communicator should not be revoked. >>> [rich] How does one then notify other ranks of the errors – does this have to be a user-level protocol ? >>> >>> >>> Looking forward, if one wants to restart the failed ranks (let’s assume we add support for this), what can be assume about the “repaired” communicator ? What can’t I assume about this communicator ? >>> >>> What you can assume depends on what is the meaning of “repaired”. Already today one can spawn new processes and reconstruct a communicator identical to the original communicator before any fault. This can be done using MPI dynamics together with the agreement available in the ULFM proposal. >>> [rich] This implies that all outstanding traffic is flushed – is this correct ? >>> >>> This is up to the MPI implementation. This is specified on the first “Advice to implementors” on the second page. >>> [rich] does not seem like a good idea – users should have guarantees on what they get if they use MPI. >>> >>> George. >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by Wesley Bland 08 Dec '13

08 Dec '13

Obviously there are various optimizations you can use if you know there will be more of fewer communicators, but in the general case, you need to include the incarnation number in the matching scheme and pick the maximum of the remaining processes. In reality, based on the way MPICH does it, if you free your communicator after you revoke it, this isn't really a problem unless you have lots and lots of transient failures. There are other ways you can do it. You can check to see if the process that sent the message is actually in the communicator. You can just throw away the context id if you aren't going to have many. You can just keep iterating the incarnation number if you aren't going to try to handle so many failures that you overflow you data type. Wesley > On Dec 8, 2013, at 7:14 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Hi Guys, > > Thanks for the detailed responses (and sorry for going off-topic). > > Does the incarnation number have to be included in matching? > > I assume that we would need to track an incarnation number for each context ID. When creating a communicator, we can't guarantee that you'll get the same incarnation number everywhere for a given Context ID. Do you select the highest incarnation number from among the processes involved? I assume the incarnation counter is meant to be monotonic increasing -- what happens when it reaches its maximum value? > > ~Jim. > > >> On Thu, Dec 5, 2013 at 1:55 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> Yes you can free the revoked communicator so you can also reuse the context id. There are a few ways you can handle this. One is to add an incarnation number to the communicator or process (which is the route chosen by the UTK ULFM implementation). Another is that once you know a process has failed, you ignore all future messages from that process (this causes issues if you want to deal with transient failures, but we’ve defined the scope of the work to exclude those). I’m sure there are more, possibly cleverer ways of solving this but those came to mind first. >> >> Wesley >> >> >>> On Dec 5, 2013, at 12:39 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>> >>> This is a little off-topic from the current discussion -- can revoking communicators could lead to leaking context IDs? After revoking a communicator, can you safely free it and then reuse the same context ID without concerns about erroneous messages arriving with that context ID? Or is that context ID dead for the remainder of the execution? >>> >>> ~Jim. >>> >>> >>>> On Thu, Dec 5, 2013 at 10:49 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >>>> I think there’s still confusion here between the current FT proposal and the roadmap for previous proposals. I think the previous proposal was designed to be a stopgap solution that would allow MPI to stabilize, but not necessarily fully recover without a lot of work. The current proposal provides all the tools necessary for both stabilization and recovery. Thus we haven’t been discussing the “next step”, because I think most of us see the next step not as a standardization effort for failure recovery, but something outside of the standard such as the libraries that will make FT easier to use. >>>> >>>> It’s true that MPI_COMM_REVOKE will make a current communicator unusable for further communication. Period. However, you can still query all of the local data about the communicator (rank, size, topology, info keys, etc.) to allow you to reconstruct a new communicator. That reconstruction is very much possible with the current proposal. It’s obviously not cheap, but I think we all know that FT recovery isn’t necessarily cheap. I think somewhere along the way we even had an example of how you could reconstruct a communicator with processes retaining their original ranks (though if you’re not going to replace the failed processes, I’m not sure what use it is to retain your rank since your algorithm will need to be able to handle such things anyway). >>>> >>>> MPI_COMM_REVOKE is the mechanism by which a process notifies other ranks of errors. No user-level protocol is necessary, though there are specific cases where a user-level solution might be a better solution depending on the communication patter. When you call MPI_COMM_REVOKE, you give up on any remaining (incoming) traffic that’s outstanding on the communicator. For remote processes, they too will not receive any messages after they return the error code MPI_COMM_REVOKED. This doesn’t preclude them from delaying reporting MPI_ERR_REVOKED while they flush any messages that have already been received, however, once the error code is returned, the communicator is dead. Users have to be able to reason about this and I don’t think it’s entirely unclear. Once revoke is called, that communicator is hosed. You can create a new communicator and use it to decide what’s necessary to recover your application (algorithm state, data repair, etc.). That’s the guarantee that users get: once the communicator is revoked, all messages/requests that have not been received are cancelled and will never be completed. Any other guarantee invites lots of trouble as you have to deal with race conditions involving who receives the revoke notification before any other messages and whether or not all of the links necessary to complete a message transfer are still available. The entire point of the REVOKE call is a last resort operation to prevent deadlock. It’s entirely possible that an application won’t need to use the revoke if there aren’t complex communication dependencies. It may be possible to take care of things at the user level in a way that’s cheaper than using REVOKE. >>>> >>>> The entire basis of this proposal is that we can’t provide a fault tolerance solution that will cheaply stabilize MPI, repair communicators, and notify all processes of failures. It’s possible to do all of these things in a way that’s not outrageously expensive, but they require cooperation from the application's perspective (or libraries that act on the application’s behalf). It’s possible to do many of those things, but they are so costly that they would never be approved. It’s much more effective to do them on top of MPI. We’re just providing the low level tools necessary to make it happen. >>>> >>>> Wesley >>>> >>>>> On Dec 5, 2013, at 8:14 AM, Richard Graham <richardg(a)mellanox.com> wrote: >>>>> >>>>> >>>>> >>>>> From: mpiwg-ft [mailto:[email protected]] On Behalf Of George Bosilca >>>>> Sent: Wednesday, November 27, 2013 3:35 PM >>>>> To: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [mpiwg-ft] MPI_Comm_revoke behavior >>>>> >>>>> >>>>> On Nov 27, 2013, at 20:54 , Richard Graham <richardg(a)mellanox.com> wrote: >>>>> >>>>> >>>>> On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> wrote: >>>>> >>>>> >>>>> >>>>> I am thinking about the next step, and have some questions on the semantics of MPI_Comm_revoke() >>>>> >>>>> What next step are you referring to? >>>>> [rich] To the full recovery stage. Post what we are talking about now. >>>>> >>>>> Full recovery stage? Can you expose a little more details here please. >>>>> [rich] the original intent was to allow for full restoration of communicators after failure, with minimal impact on those ranks that did not fail (don’t want to get into what that means now …). Those goals were reduced for pragmatic reasons. I want to make sure that when/if there is work continued in this direction, the current proposal does not preclude this. One of the issues raised to me recently is that after a revoke one will not be able to accomplish such a goal on the remaining ranks – e.g., ranks will be reassigned. I am following up very specifically on this question. >>>>> >>>>> >>>>> >>>>> >>>>> - When the routine returns, can the communicator ever be used again ? If I remember correctly, the communicator is available for point-to-point traffic, but not collective traffic – is this correct ? >>>>> >>>>> A revoked communicator is unable to support any communication (point-to-point or collective) with the exception of agree and shrink. If this is not clear enough in the current version of the proposal we should definitively address it. >>>>> [rich] does this mean all current state (aside from who is alive) associated with the communicator is gone ? >>>>> >>>>> Every deterministic information is still available (info and attributes). You can look for the group of processes associated with the communicator, as well as the group of failed. If what you are looking for is the possible unexpected messages, this is up to the implementation (see below). >>>>> [rich] don’t understand >>>>> >>>>> Can’t rely on continuing sending pending messages ? >>>>> >>>>> Not on a revoked communicator. If continuing to exchange messages is a requirement, the communicator should not be revoked. >>>>> [rich] How does one then notify other ranks of the errors – does this have to be a user-level protocol ? >>>>> >>>>> >>>>> Looking forward, if one wants to restart the failed ranks (let’s assume we add support for this), what can be assume about the “repaired” communicator ? What can’t I assume about this communicator ? >>>>> >>>>> What you can assume depends on what is the meaning of “repaired”. Already today one can spawn new processes and reconstruct a communicator identical to the original communicator before any fault. This can be done using MPI dynamics together with the agreement available in the ULFM proposal. >>>>> [rich] This implies that all outstanding traffic is flushed – is this correct ? >>>>> >>>>> This is up to the MPI implementation. This is specified on the first “Advice to implementors” on the second page. >>>>> [rich] does not seem like a good idea – users should have guarantees on what they get if they use MPI. >>>>> >>>>> George. >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpiwg-ft mailing list >>>>> mpiwg-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by Jim Dinan 08 Dec '13

08 Dec '13

Hi Guys, Thanks for the detailed responses (and sorry for going off-topic). Does the incarnation number have to be included in matching? I assume that we would need to track an incarnation number for each context ID. When creating a communicator, we can't guarantee that you'll get the same incarnation number everywhere for a given Context ID. Do you select the highest incarnation number from among the processes involved? I assume the incarnation counter is meant to be monotonic increasing -- what happens when it reaches its maximum value? ~Jim. On Thu, Dec 5, 2013 at 1:55 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > Yes you can free the revoked communicator so you can also reuse the > context id. There are a few ways you can handle this. One is to add an > incarnation number to the communicator or process (which is the route > chosen by the UTK ULFM implementation). Another is that once you know a > process has failed, you ignore all future messages from that process (this > causes issues if you want to deal with transient failures, but we’ve > defined the scope of the work to exclude those). I’m sure there are more, > possibly cleverer ways of solving this but those came to mind first. > > Wesley > > > On Dec 5, 2013, at 12:39 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > This is a little off-topic from the current discussion -- can revoking > communicators could lead to leaking context IDs? After revoking a > communicator, can you safely free it and then reuse the same context ID > without concerns about erroneous messages arriving with that context ID? > Or is that context ID dead for the remainder of the execution? > > ~Jim. > > > On Thu, Dec 5, 2013 at 10:49 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> I think there’s still confusion here between the current FT proposal and >> the roadmap for previous proposals. I think the previous proposal was >> designed to be a stopgap solution that would allow MPI to stabilize, but >> not necessarily fully recover without a lot of work. The current proposal >> provides all the tools necessary for both stabilization and recovery. Thus >> we haven’t been discussing the “next step”, because I think most of us see >> the next step not as a standardization effort for failure recovery, but >> something outside of the standard such as the libraries that will make FT >> easier to use. >> >> It’s true that MPI_COMM_REVOKE will make a current communicator unusable >> for further communication. Period. However, you can still query all of the >> local data about the communicator (rank, size, topology, info keys, etc.) >> to allow you to reconstruct a new communicator. That reconstruction is very >> much possible with the current proposal. It’s obviously not cheap, but I >> think we all know that FT recovery isn’t necessarily cheap. I think >> somewhere along the way we even had an example of how you could reconstruct >> a communicator with processes retaining their original ranks (though if >> you’re not going to replace the failed processes, I’m not sure what use it >> is to retain your rank since your algorithm will need to be able to handle >> such things anyway). >> >> MPI_COMM_REVOKE is the mechanism by which a process notifies other ranks >> of errors. No user-level protocol is necessary, though there are specific >> cases where a user-level solution might be a better solution depending on >> the communication patter. When you call MPI_COMM_REVOKE, you give up on any >> remaining (incoming) traffic that’s outstanding on the communicator. For >> remote processes, they too will not receive any messages after they return >> the error code MPI_COMM_REVOKED. This doesn’t preclude them from delaying >> reporting MPI_ERR_REVOKED while they flush any messages that have already >> been received, however, once the error code is returned, the communicator >> is dead. Users have to be able to reason about this and I don’t think it’s >> entirely unclear. Once revoke is called, that communicator is hosed. You >> can create a new communicator and use it to decide what’s necessary to >> recover your application (algorithm state, data repair, etc.). That’s the >> guarantee that users get: once the communicator is revoked, all >> messages/requests that have not been received are cancelled and will never >> be completed. Any other guarantee invites lots of trouble as you have to >> deal with race conditions involving who receives the revoke notification >> before any other messages and whether or not all of the links necessary to >> complete a message transfer are still available. The entire point of the >> REVOKE call is a last resort operation to prevent deadlock. It’s entirely >> possible that an application won’t need to use the revoke if there aren’t >> complex communication dependencies. It may be possible to take care of >> things at the user level in a way that’s cheaper than using REVOKE. >> >> The entire basis of this proposal is that we can’t provide a fault >> tolerance solution that will cheaply stabilize MPI, repair communicators, >> and notify all processes of failures. It’s possible to do all of these >> things in a way that’s not outrageously expensive, but they require >> cooperation from the application's perspective (or libraries that act on >> the application’s behalf). It’s possible to do many of those things, but >> they are so costly that they would never be approved. It’s much more >> effective to do them on top of MPI. We’re just providing the low level >> tools necessary to make it happen. >> >> Wesley >> >> On Dec 5, 2013, at 8:14 AM, Richard Graham <richardg(a)mellanox.com> wrote: >> >> >> >> *From:* mpiwg-ft [mailto:[email protected]<mpiwg-ft-bounces(a)lists.mpi-forum.org> >> ] *On Behalf Of *George Bosilca >> *Sent:* Wednesday, November 27, 2013 3:35 PM >> *To:* MPI WG Fault Tolerance and Dynamic Process Control working Group >> *Subject:* Re: [mpiwg-ft] MPI_Comm_revoke behavior >> >> >> >> On Nov 27, 2013, at 20:54 , Richard Graham <richardg(a)mellanox.com> wrote: >> >> >> On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> >> wrote: >> >> >> >> I am thinking about the next step, and have some questions on the >> semantics of MPI_Comm_revoke() >> >> What next step are you referring to? >> >> [rich] To the full recovery stage. Post what we are talking about now. >> >> >> Full recovery stage? Can you expose a little more details here please. >> >> [rich] the original intent was to allow for full restoration of >> communicators after failure, with minimal impact on those ranks that did >> not fail (don’t want to get into what that means now …). Those goals were >> reduced for pragmatic reasons. I want to make sure that when/if there is >> work continued in this direction, the current proposal does not preclude >> this. One of the issues raised to me recently is that after a revoke one >> will not be able to accomplish such a goal on the remaining ranks – e.g., >> ranks will be reassigned. I am following up very specifically on this >> question. >> >> >> >> >> >> - When the routine returns, can the communicator ever be used >> again ? If I remember correctly, the communicator is available for >> point-to-point traffic, but not collective traffic – is this correct ? >> >> A revoked communicator is unable to support any communication >> (point-to-point or collective) with the exception of agree and shrink. If >> this is not clear enough in the current version of the proposal we should >> definitively address it. >> >> [rich] does this mean all current state (aside from who is alive) >> associated with the communicator is gone ? >> >> >> Every deterministic information is still available (info and attributes). >> You can look for the group of processes associated with the communicator, >> as well as the group of failed. If what you are looking for is the possible >> unexpected messages, this is up to the implementation (see below). >> [rich] don’t understand >> >> >> Can’t rely on continuing sending pending messages ? >> >> >> Not on a revoked communicator. If continuing to exchange messages is a >> requirement, the communicator should not be revoked. >> [rich] How does one then notify other ranks of the errors – does this >> have to be a user-level protocol ? >> >> >> >> Looking forward, if one wants to restart the failed ranks >> (let’s assume we add support for this), what can be assume about the >> “repaired” communicator ? What can’t I assume about this communicator ? >> >> What you can assume depends on what is the meaning of “repaired”. Already >> today one can spawn new processes and reconstruct a communicator identical >> to the original communicator before any fault. This can be done using MPI >> dynamics together with the agreement available in the ULFM proposal. >> >> [rich] This implies that all outstanding traffic is flushed – is this >> correct ? >> >> >> This is up to the MPI implementation. This is specified on the first >> “Advice to implementors” on the second page. >> [rich] does not seem like a good idea – users should have guarantees on >> what they get if they use MPI. >> >> George. >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by George Bosilca 06 Dec '13

06 Dec '13

The WG group has regular calls that are widely publicized and open to anyone interested. I propose that in the future we strongly suggest to interested parties to join the WG calls to have their concerns/questions addresses firsthand, directly by the people active in the WG. We would have loved to hear any expert express her/his concerns in a live discussion. George. On Dec 6, 2013, at 15:15 , Richard Graham <richardg(a)mellanox.com> wrote: > This was raised as a concern at SC be an expert in the field, and specifically the issue of preserving rank I'd. We just need to follow up and ensure there is not a misunderstanding. > > Rich > > ------Original Message------ > From: Wesley Bland > To: MPI WG Fault Tolerance and Dynamic Process Control working Group > Cc: MPI WG Fault Tolerance and Dynamic Process Control working Group > ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [mpiwg-ft] MPI_Comm_revoke behavior > Sent: Dec 6, 2013 9:08 AM > > Rich, > This is something we've discussed many times on the con calls and mailing list, but we can discuss it on Monday as well. Aurélien will also be presenting slides during the FT plenary time demonstrating sample use cases. We haven't yet come up with something that we'll be excluding with the current proposal. > Wesley > On Dec 6, 2013, at 7:47 AM, Richard Graham <richardg(a)mellanox.com> wrote: > > I would disagree with your characterization of the previous approach as anything but minimalistic. > > Let’s talk about this in the WG slot on Monday. I have to say that in some way I totally missed the point that this is intended to “be it”, so need to carefully re-evaluate the proposal in that light. My main concern is that the standard is supposed to provide a means for supporting a broad range of FT methodologies on top of this. Need to make sure that some of the approaches people do want to take are being blocked. Also, concern had been expressed that the resulting behavior will prevent many users from using it, so need to talk through these issues (I will say that the behavior described to me is a show stopper for many users, so need to make sure there is not a misunderstanding). > > Rich > > From: mpiwg-ft [mailto:[email protected]] On Behalf Of George Bosilca > Sent: Thursday, December 05, 2013 11:38 AM > To: MPI WG Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [mpiwg-ft] MPI_Comm_revoke behavior > > > On Dec 5, 2013, at 15:14 , Richard Graham <richardg(a)mellanox.com> wrote: > > > > [rich] the original intent was to allow for full restoration of communicators after failure, with minimal impact on those ranks that did not fail (don’t want to get into what that means now …). Those goals were reduced for pragmatic reasons. > > The goals were not reduced, ULFM is a completely new approach based on a pragmatic design. To emphasize what Wesley suggested, ULFM is not an all-encompassing solution (unlike previous proposals). Instead is a __minimalistic__ set of building blocks for stabilization and recovery allowing the construction of more complex FT mechanism. So far, the exploration of such complementary FT approaches have remained in the real of research, outside the WG > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by Richard Graham 06 Dec '13

06 Dec '13

Great ------Original Message------ From: Aurélien Bouteiller To: MPI WG Fault Tolerance and Dynamic Process Control working Group ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] MPI_Comm_revoke behavior Sent: Dec 6, 2013 9:17 AM It certainly is a valid and important issue to be able to restore a deployment with rank isomorphism. Gladly the requirement that this scenario would be supported has been accounted for from day 1. It is one of the simple use cases that is already deployed by many users. I’ll present code snippets on monday. Aurelien Le 6 déc. 2013 à 09:15, Richard Graham <richardg(a)mellanox.com> a écrit : > This was raised as a concern at SC be an expert in the field, and specifically the issue of preserving rank I'd. We just need to follow up and ensure there is not a misunderstanding. > > Rich > > ------Original Message------ > From: Wesley Bland > To: MPI WG Fault Tolerance and Dynamic Process Control working Group > Cc: MPI WG Fault Tolerance and Dynamic Process Control working Group > ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [mpiwg-ft] MPI_Comm_revoke behavior > Sent: Dec 6, 2013 9:08 AM > > Rich, > This is something we've discussed many times on the con calls and mailing list, but we can discuss it on Monday as well. Aurélien will also be presenting slides during the FT plenary time demonstrating sample use cases. We haven't yet come up with something that we'll be excluding with the current proposal. > Wesley > On Dec 6, 2013, at 7:47 AM, Richard Graham <richardg(a)mellanox.com> wrote: > > I would disagree with your characterization of the previous approach as anything but minimalistic. > > Let’s talk about this in the WG slot on Monday. I have to say that in some way I totally missed the point that this is intended to “be it”, so need to carefully re-evaluate the proposal in that light. My main concern is that the standard is supposed to provide a means for supporting a broad range of FT methodologies on top of this. Need to make sure that some of the approaches people do want to take are being blocked. Also, concern had been expressed that the resulting behavior will prevent many users from using it, so need to talk through these

1 0