July 2013 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Biweekly phone meeting
by Aurélien Bouteiller 23 Jul '13

23 Jul '13

Dear WG members, This is a reminder that according to our planning, we are having our regular phone meeting tuesday at 3pm EDT. Date: July 23, Time: 3pm EDT/New York Dial-in information: 218-339-4600 Code: 623998# Next Meetings: * August. 6, 2013 * August. 20, 2013 -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [Mpi3-ft] Distinguishing errors from failures
by Aurélien Bouteiller 17 Jul '13

17 Jul '13

Le 17 juil. 2013 ï¿½ 12:16, Jim Dinan <james.dinan(a)gmail.com> a ï¿½crit : > Hi George and Aurilien, > > Thanks for the detailed responses. I looked at the paper, and it indicates that failure detection is needed when ANY_SOURCE is used, and I assume also when passive target RMA is used, since a process can fail while holding the lock. Won't this have an impact on performance? > Depending on implementation it may. But it can be mitigated to pretty much nothing in practice: For ANY_SOURCE, the code can go directly into optimized mode, w/o failure detection for "some time". After "some time (implementation dependent)" has passed and the implementation suspects this is not normal, actions to detect potential failures can be taken (interrogate a failure detector, poke around to see who's still responding, etc). The result is that latency w/o failure is not impacted when it matters, detection latency is added only for messages that already have long latency (due to load imbalance in the user code, presumably). Bandwidth is never an issue. Once a message is matched, it is not "any source" anymore. For lock, we discussed together of how the typical implementation can work in practice w/o adding much in terms of failure detection. Another implementation may be more problematic, but a similar approach as for ANY_SOURCE could be taken. If one wants to favor raw performance, it is always possible to do so at the expense of failure detection delay. Also note that in the paper, we do have the (optional) failure detection framework enabled at all time during the measurements, and performance are unchanged nonetheless. Also, many HPC fabrics embed advanced failure detection and reporting. We do not mandate that they are present, but these would provide both fairly accurate detection and good response time. Aurelien > ~Jim. > > > On Tue, Jul 16, 2013 at 7:27 PM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > In addition to Aurelien's answer, there is something else that I think should be stressed in this context, something I feel the WG failed to make clear enough in its interactions with the forum. > > There is one single strong mandate from an MPI implementation in the current FT proposal: a revoked communicator is improper for further communications. This is about the extent imposed on MPI implementations by the current proposal, both in terms of capabilities and overheads. And still, there were complaints raised about it ï¿½ > > Why is this so? Because there is no mandatory error detection and propagation, when the MPI library detects a failure only local dispatch of this information is necessary. Of course, one would expect a high quality MPI implementation to do it's best to ensure a certain level of quality of services here, a significantly lesser effort than ensuring some other "high quality" type of capabilities (namely progress and fairness). The paper mentioned by Aurelien proves that at least under certain assumptions this can be achieved with minimal/unnoticeable overhead. > > >From there, the application itself is responsible to make good use of the functions provided by the FT proposal to handle the failure in a meaningful way for the application (again there is no imposed FT model). MPI_Comm_revoke provides a communication channel where at the request of the application the knowledge about a process failure is propagated to other MPI processes. The ACK function to locally acknowledge the failure. The agreement to reach a consensus for building more complex FT methods, and finally shrink to take you back to a communicator that has all the properties of a sane/workable MPI communicator. > > George. > > On Jul 16, 2013, at 23:41 , Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> wrote: > > > Jim, > > > > There is no specific cost to enable FT at all time, so this might not be such a crucial issue actually, since the initial assumption is not true. > > > > You can point to the following paper that investigates the cost sustained by applications and stressful micro benchmarks to support this claim with solid evidence. > > > > Bland, W., Bouteiller, A., Herault, T., Hursey, J., Bosilca, G., Dongarra, J.J. ï¿½An Evaluation of User-Level Failure Mitigation Support in MPI,ï¿½ Computing, Springer, 2013, issn 0010-4885X, http://dx.doi.org/10.1007/s00607-013-0331-3 > > > > The intuitive explanation to the no-impact result is that the spec does not change the behavior of any existing MPI functions. If a function has local completion, it retains local completion even when it reports a failure. It merely adds a new class of errors to let users know that what killed them is process failure, rather than say network retransmit error. > > > > The FT additions kick in only -after- a failure was reported. Nothing is implicit. The MPI implementation is not expected to "fix" things on its own in the background, normal MPI functions are not overloaded with failure related actives. All recovery actions are triggered by explicit calls from the user code, which removes all potential performance problems from "spilling" recovery activity inside failure-free path. > > > > All this means that the codebase (see the prototype implementation) stays basically unchanged, with no modifications in the transport layer, no modifications in the collective framework, etc. > > > > ~~~~~~ > > > > As you noted, it can also be turned on/off by mpirun switches if deemed necessary. It is standard compliant (and this is deliberate) to have all FT recovery routines map to no-op (revoke=no-op) or no-FT equivalents (agree=allreduce) when no fault tolerance is needed, and even FT codes will run fine on such an MPI library (as long as there are no failures, of course). If some implementation really wants to "optimize out" anything FT related (or even load-in only if required), This would be our recommendation. But again, the performance gain is expected to be minor, if even measurable. > > > > > > Aurelien > > > > > > Le 16 juil. 2013 ï¿½ 16:23, Jim Dinan <james.dinan(a)gmail.com> a ï¿½crit : > > > >> Hi FT WG, > >> > >> I am doing my best to socialize the FT proposal at Intel and gathered a piece of feedback to bring back to the WG. > >> > >> There was a concern that any time the user registers an error handler, fault tolerance could be "switched on" because MPI_Comm_set_errhandler() does not distinguish between error classes. The assumption was that, when switched on, there would be space/time costs associated with fault tolerance. How does the current proposal determine when fault tolerance should be enabled? > >> > >> One suggested mechanism was to add a function, MPI_Comm_set_faulthandler() that allows the programmer to distinguish between errors and failures. This would allow the runtime to determine when fault tolerance was desired. I think the way this is handled currently is to rely on the implementation switching on/off fault tolerance when the job is launched. > >> > >> ~Jim. > >> _______________________________________________ > >> mpi3-ft mailing list > >> mpi3-ft(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > -- > > * Dr. Aurï¿½lien Bouteiller > > * Researcher at Innovative Computing Laboratory > > * University of Tennessee > > * 1122 Volunteer Boulevard, suite 309b > > * Knoxville, TN 37996 > > * 865 974 9375 > > > > > > > > > > > > > > > > > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

[Mpi3-ft] Re: WG call on tuesday aug. 9, 3pm est
by Wesley Bland 17 Jul '13

17 Jul '13

On July 16, 2013 at 6:25:46 PM, George Bosilca (bosilca(a)icl.utk.edu) wrote: Jim, On Jul 16, 2013, at 22:09 , Jim Dinan <james.dinan(a)gmail.com> wrote: Hi George, Thanks for the detailed response. Let's set aside the MPI_Comm_create_group discussion. You raised a good point about the difficulty in reaching consensus on the group argument that has not come up before. If there is some mechanism for consensus, then Comm_create_group will just work. We can hardly settle here. I was not trying to pinpoint the difficulty of using MPI_Comm_create_group to build a new communicator, I was gently trying to suggest the impossibility of doing so with only the MPI_Comm_create_group call. Using MPI_Comm_agree to agree that the creation of the communicator based on the group (MPI_Comm_create_group) was successful requires you to have a __valid__ (potentially revoked) communicator to do so. You can't use the old communicator, as you will not be able to make a difference between the processes that failed before and after the MPI_Comm_create_group. Similarly, you can't use the newly created communicator either, as it might be incomplete (you did not have an agreement on the group of dead processes on the first place). However, as I mentioned before, there are scenarios where you can use this function if complemented with a shrink (that will in fact build both the consistent knowledge about the dead processes and a communicator to further agree on). I still have some remaining concerns about supporting a usage model where the communicator has "holes" in it. The group query routine that I was thinking of is a local query, which would return information on which processes I know to have failed. Any particular reason the current get_acked() is not satisfying? An application won't be able to track failures that occur within libraries, Within libraries? so this query is necessary to find out what is locally known. I assume that MPI will have to track this information anyway, in case you attempt to communicate with a failed process. There is no such requirement from the MPI library in the current proposal. In many applications global knowledge is not required, so there is no reason to propagate such information and force the associated overhead on the application. In these applications the failure will eventually be discovered if a communication with the dead process is initiated. Consensus. Is it true that right now, that the only ways to reach consensus on the failed group is to revoke the communicator, shrink it, and then compare groups, or to implement your own consensus protocol on top of send/recv? The concern I have is that the revoke method requires the application to destroy the old communicator and renumber the processes, which the programmer may not want to do. The ranks could have some important meaning to the application. As I hinted in my previous email there are many other ways to turn around this problem. You can use topologies, or you can use a shadow communicator that will deal with the revoke operations, while your "working" communicator (which is meant only for point-to-point communications) remains "unrevoked" (but eventually the failures will be acknowledged). I am probably going to regret asking this, but is it possible to include an MPI_Comm_resume() function that re-activates a revoked communicator with holes in it? For many reasons related to the complexities of distributed systems (lack of synchronization in the error detection, divergent view of the entire system from each process) this operation must have a consensus meaning. Thus in terms of cost it is similar to MPI_Comm_shrink (except the reordering of the processes). It might provide some limited benefit, for people that want to use such type of scenario. Now, if by doing the re-enable you expect that the communicator will behave as a freshly new communicator and everything MPI-related, file, one-sided, collective will just work on this communicator with holes … then we're talking about something so complex that I would not even dare considering for inclusion in the standard. I don't think that was the intent here. I think the intent was to essentially put the communicator into the same state as is was before the revoke, where the difference is that now all of the processes in the communicator know about the failure. This means that they can continue pt2pt, cannot use collectives, can use wildcards (if they've called FAILURE_ACK), and whatever else we say you can do after a failure. The use case we're trying to enable is to say that you can notify the other processes of a failure (global failure knowledge and the deadlock prevention that comes with it) without having to create a new communicator. Yes, it will be as expensive as revoking and shrinking the communicator, but the difference is that you don't have to shrink the communicator to make it work. If you want a communicator with holes, you use your existing communicator. If you want a communicator without holes, you shrink (or call COMM_CREATE_GROUP). Wesley George. ~Jim. On Mon, Jul 15, 2013 at 9:28 AM, George Bosilca <bosilca(a)icl.utk.edu> wrote: Jim, let me try to articulate a comprehensible answer without going too deep into technicalities. On Jul 10, 2013, at 16:07 , Jim Dinan <james.dinan(a)gmail.com> wrote: Hi George, Re: comm_create and friends -- I'd be interested in hearing the result of the F2F discussion. I've been the person complaining about this, so if it would be helpful for me to join you guys over the phone next Friday, let me know. I did not have the opportunity to hear your complaint about this. Can you please summarize it here on the mailing list to make sure 1) that we are all at the same level of understanding; 2) the complaint reach all the interested audience and 3) we have the opportunity to address it as accurately as possible. I'm concerned that the current spec is missing a few features that are needed for the usage model where users continue to use a communicator with "holes" in it. In order for this model to work well, users must be able to query which processes are known to have failed in a given communicator. The proposed FT framework was designed to address a wide spectrum of possible approaches, by covering two extreme cases, on one side where global knowledge and global actions are required and on the other side where only local knowledge and local actions are required, while providing building blocks for all other intermediary approaches. Looking specifically at your concern it seems that you want to continue to use a communicator with holes while avoiding the revoke, but expect to need global knowledge about the failures. Thus you propose to have a special call GET_ALL_FAILED that provides you the list of failed processes. Interesting idea but unfortunately it has pitfalls. - In order to be able to use the list of failed processes to create new communicators using the COMM_CREATE_GROUP this list should be identical on all processes. Therefore the call to GET_ALL_FAILED is similar to an agreement (collective and must have a consensus meaning). Then the obvious question will be how do you synchronize all the living processes to call this blocking function together? This is why you need an atomic broadcast (guaranteed delivery despite failure) that can interrupt the normal behavior of the application. And this is typically the goal of the revoke functionality. You could implement your own protocol of failure knowledge propagation inside your communication scheme, but imagine how much your code will be impacted by this: at any point where you do a communication, you must be able to receive either a normal message of your application, or a notification message. without the revoke functionality (the knowledge about the fault is only propagated to processes that directly communicate with the dead nodes)? Do you really intend to implement your own protocol of failure knowledge propagation integrated in the application? Given the current interface, users are not able to query the set of failed processes without creating a new communicator and translating ranks, via MPI_Comm_shrink. This requires the user to first revoke the parent communicator, which is something we want to avoid. Again the revoke is an optional step. If you have a special way to have a consensus over all still alive processes about the list of dead processes, you just have to use it. However, for the sake of performance and portability we strongly believe that MPI should propose such a functionality for the users, that unlike you, don't want to implement their own consensus. Along these same lines, we should also ensure that MPI_Comm_create_group will work as expected when a communicator with holes is used, provided the output group excludes failed processes (i.e., the operation is not collective over any failed processes). I fail to see your concern here as I can't imagine which of the part of the current proposal prevent you from using such an approach. So from a technical point of view this approach should work in the context of the current proposal Now from a performance point of view the story is slightly different. While in the case of MPI_Comm_shrink highly optimized implementations have the opportunity for merge all the stage of the operation (consensus over the dead processes and creation of a new communicator) in a single step, your approach will have to validate (via an agreement) that the newly created communicator is indeed valid (aka, no new failure has been discovered during the communicator creation). Given the current interface, users are not able to query the set of failed processes without creating a new communicator and translating ranks, via MPI_Comm_shrink. This requires the user to first revoke the parent communicator, which is something we want to avoid. Again the revoke is an optional step. If you have a special way to have a consensus over all still alive processes about the list of dead processes, you just have to use it. However, for the sake of performance and portability we strongly believe that MPI should propose such a functionality for the users, that unlike you, don't want to implement their own consensus. Along these same lines, we should also ensure that MPI_Comm_create_group will work as expected when a communicator with holes is used, provided the output group excludes failed processes (i.e., the operation is not collective over any failed processes). I fail to see your concern here as I can't imagine which of the part of the current proposal prevent you from using such an approach. So from a technical point of view this approach should work in the context of the current proposal Now from a performance point of view the story is slightly different. While in the case of MPI_Comm_shrink highly optimized implementations have the opportunity for merge all the stage of the operation (consensus over the dead processes and creation of a new communicator) in a single step, your approach will have to validate (via an agreement) that the newly created communicator is indeed valid (aka, no new failure has been discovered during the communicator creation). And to do so, you will have to have a communicator without holes, because agree will not provide you appropriate information on communicators with holes. This might not require any text changes, unless we want to allow this operation on revoked communicators. At this point I get confused as I was under the impression that your goal was to avoid revoking the communicator? Anyway, we do not want original meaning of COMM_CREATE_GROUP to be tainted by FT semantic. Especially true for comm_create_group which is supposed to be super scalable. See above for why it serves little purpose anyway, if it is not to become an expensive agreement operation. George. Re: Roadmap -- Before a reading, it might be helpful to give a brief presentation to the Forum again giving the high level ideas and justifications for each new addition to the FT proposal. I think it's been long enough that people have forgotten the details and this might help them feel more comfortable that the proposal is complete and self-consistent. There was at least one [more or less] "brief" presentation of the FT proposal at every meeting for the last two years. I would even emphasize the fact that over the last year no major modification of the proposal has been put forward, fact that might indicate a certain level of completeness and self-consistency. I'm just trying to convey the temperature in the room, as I felt it. I think a 15 minute, very high level warm-up immediately before the reading on the usage models, big ideas, and conventions would go a long way toward prepping the audience. ~Jim. _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by Jim Dinan 17 Jul '13

17 Jul '13

Hi George, Responses below: > The group query routine that I was thinking of is a local query, which > would return information on which processes I know to have failed. > > > Any particular reason the current get_acked() is not satisfying? > Eek, ok, I had completely forgotten about this function. This is what I was asking for, hopefully my previous message now makes more sense. :) > I am probably going to regret asking this, but is it possible to include > an MPI_Comm_resume() function that re-activates a revoked communicator with > holes in it? > > > For many reasons related to the complexities of distributed systems (lack > of synchronization in the error detection, divergent view of the entire > system from each process) this operation must have a consensus meaning. > Thus in terms of cost it is similar to MPI_Comm_shrink (except the > reordering of the processes). It might provide some limited benefit, for > people that want to use such type of scenario. Now, if by doing the > re-enable you expect that the communicator will behave as a freshly new > communicator and everything MPI-related, file, one-sided, collective will > just work on this communicator with holes … then we're talking about > something so complex that I would not even dare considering for inclusion > in the standard. > What if MPI_COMM_RESUME duplicated the revoked communicator and left the holes in place? This would preserve the revoked semantic, but allow the user to continue without renumbering the processes. ~Jim.

1 0

Re: [Mpi3-ft] Distinguishing errors from failures
by Jim Dinan 17 Jul '13

17 Jul '13

Hi George and Aurilien, Thanks for the detailed responses. I looked at the paper, and it indicates that failure detection is needed when ANY_SOURCE is used, and I assume also when passive target RMA is used, since a process can fail while holding the lock. Won't this have an impact on performance? ~Jim. On Tue, Jul 16, 2013 at 7:27 PM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > In addition to Aurelien's answer, there is something else that I think > should be stressed in this context, something I feel the WG failed to make > clear enough in its interactions with the forum. > > There is one single strong mandate from an MPI implementation in the > current FT proposal: a revoked communicator is improper for further > communications. This is about the extent imposed on MPI implementations by > the current proposal, both in terms of capabilities and overheads. And > still, there were complaints raised about it … > > Why is this so? Because there is no mandatory error detection and > propagation, when the MPI library detects a failure only local dispatch of > this information is necessary. Of course, one would expect a high quality > MPI implementation to do it's best to ensure a certain level of quality of > services here, a significantly lesser effort than ensuring some other "high > quality" type of capabilities (namely progress and fairness). The paper > mentioned by Aurelien proves that at least under certain assumptions this > can be achieved with minimal/unnoticeable overhead. > > >From there, the application itself is responsible to make good use of the > functions provided by the FT proposal to handle the failure in a meaningful > way for the application (again there is no imposed FT model). > MPI_Comm_revoke provides a communication channel where at the request of > the application the knowledge about a process failure is propagated to > other MPI processes. The ACK function to locally acknowledge the failure. > The agreement to reach a consensus for building more complex FT methods, > and finally shrink to take you back to a communicator that has all the > properties of a sane/workable MPI communicator. > > George. > > On Jul 16, 2013, at 23:41 , Aurélien Bouteiller <bouteill(a)icl.utk.edu> > wrote: > > > Jim, > > > > There is no specific cost to enable FT at all time, so this might not be > such a crucial issue actually, since the initial assumption is not true. > > > > You can point to the following paper that investigates the cost > sustained by applications and stressful micro benchmarks to support this > claim with solid evidence. > > > > Bland, W., Bouteiller, A., Herault, T., Hursey, J., Bosilca, G., > Dongarra, J.J. “An Evaluation of User-Level Failure Mitigation Support in > MPI,“ Computing, Springer, 2013, issn 0010-4885X, > http://dx.doi.org/10.1007/s00607-013-0331-3 > > > > The intuitive explanation to the no-impact result is that the spec does > not change the behavior of any existing MPI functions. If a function has > local completion, it retains local completion even when it reports a > failure. It merely adds a new class of errors to let users know that what > killed them is process failure, rather than say network retransmit error. > > > > The FT additions kick in only -after- a failure was reported. Nothing is > implicit. The MPI implementation is not expected to "fix" things on its own > in the background, normal MPI functions are not overloaded with failure > related actives. All recovery actions are triggered by explicit calls from > the user code, which removes all potential performance problems from > "spilling" recovery activity inside failure-free path. > > > > All this means that the codebase (see the prototype implementation) > stays basically unchanged, with no modifications in the transport layer, no > modifications in the collective framework, etc. > > > > ~~~~~~ > > > > As you noted, it can also be turned on/off by mpirun switches if deemed > necessary. It is standard compliant (and this is deliberate) to have all FT > recovery routines map to no-op (revoke=no-op) or no-FT equivalents > (agree=allreduce) when no fault tolerance is needed, and even FT codes will > run fine on such an MPI library (as long as there are no failures, of > course). If some implementation really wants to "optimize out" anything FT > related (or even load-in only if required), This would be our > recommendation. But again, the performance gain is expected to be minor, if > even measurable. > > > > > > Aurelien > > > > > > Le 16 juil. 2013 à 16:23, Jim Dinan <james.dinan(a)gmail.com> a écrit : > > > >> Hi FT WG, > >> > >> I am doing my best to socialize the FT proposal at Intel and gathered a > piece of feedback to bring back to the WG. > >> > >> There was a concern that any time the user registers an error handler, > fault tolerance could be "switched on" because MPI_Comm_set_errhandler() > does not distinguish between error classes. The assumption was that, when > switched on, there would be space/time costs associated with fault > tolerance. How does the current proposal determine when fault tolerance > should be enabled? > >> > >> One suggested mechanism was to add a function, > MPI_Comm_set_faulthandler() that allows the programmer to distinguish > between errors and failures. This would allow the runtime to determine > when fault tolerance was desired. I think the way this is handled > currently is to rely on the implementation switching on/off fault tolerance > when the job is launched. > >> > >> ~Jim. > >> _______________________________________________ > >> mpi3-ft mailing list > >> mpi3-ft(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > -- > > * Dr. Aurélien Bouteiller > > * Researcher at Innovative Computing Laboratory > > * University of Tennessee > > * 1122 Volunteer Boulevard, suite 309b > > * Knoxville, TN 37996 > > * 865 974 9375 > > > > > > > > > > > > > > > > > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Distinguishing errors from failures
by George Bosilca 16 Jul '13

16 Jul '13

In addition to Aurelien's answer, there is something else that I think should be stressed in this context, something I feel the WG failed to make clear enough in its interactions with the forum. There is one single strong mandate from an MPI implementation in the current FT proposal: a revoked communicator is improper for further communications. This is about the extent imposed on MPI implementations by the current proposal, both in terms of capabilities and overheads. And still, there were complaints raised about it ï¿½ Why is this so? Because there is no mandatory error detection and propagation, when the MPI library detects a failure only local dispatch of this information is necessary. Of course, one would expect a high quality MPI implementation to do it's best to ensure a certain level of quality of services here, a significantly lesser effort than ensuring some other "high quality" type of capabilities (namely progress and fairness). The paper mentioned by Aurelien proves that at least under certain assumptions this can be achieved with minimal/unnoticeable overhead. >From there, the application itself is responsible to make good use of the functions provided by the FT proposal to handle the failure in a meaningful way for the application (again there is no imposed FT model). MPI_Comm_revoke provides a communication channel where at the request of the application the knowledge about a process failure is propagated to other MPI processes. The ACK function to locally acknowledge the failure. The agreement to reach a consensus for building more complex FT methods, and finally shrink to take you back to a communicator that has all the properties of a sane/workable MPI communicator. George. On Jul 16, 2013, at 23:41 , Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> wrote: > Jim, > > There is no specific cost to enable FT at all time, so this might not be such a crucial issue actually, since the initial assumption is not true. > > You can point to the following paper that investigates the cost sustained by applications and stressful micro benchmarks to support this claim with solid evidence. > > Bland, W., Bouteiller, A., Herault, T., Hursey, J., Bosilca, G., Dongarra, J.J. ï¿½An Evaluation of User-Level Failure Mitigation Support in MPI,ï¿½ Computing, Springer, 2013, issn 0010-4885X, http://dx.doi.org/10.1007/s00607-013-0331-3 > > The intuitive explanation to the no-impact result is that the spec does not change the behavior of any existing MPI functions. If a function has local completion, it retains local completion even when it reports a failure. It merely adds a new class of errors to let users know that what killed them is process failure, rather than say network retransmit error. > > The FT additions kick in only -after- a failure was reported. Nothing is implicit. The MPI implementation is not expected to "fix" things on its own in the background, normal MPI functions are not overloaded with failure related actives. All recovery actions are triggered by explicit calls from the user code, which removes all potential performance problems from "spilling" recovery activity inside failure-free path. > > All this means that the codebase (see the prototype implementation) stays basically unchanged, with no modifications in the transport layer, no modifications in the collective framework, etc. > > ~~~~~~ > > As you noted, it can also be turned on/off by mpirun switches if deemed necessary. It is standard compliant (and this is deliberate) to have all FT recovery routines map to no-op (revoke=no-op) or no-FT equivalents (agree=allreduce) when no fault tolerance is needed, and even FT codes will run fine on such an MPI library (as long as there are no failures, of course). If some implementation really wants to "optimize out" anything FT related (or even load-in only if required), This would be our recommendation. But again, the performance gain is expected to be minor, if even measurable. > > > Aurelien > > > Le 16 juil. 2013 ï¿½ 16:23, Jim Dinan <james.dinan(a)gmail.com> a ï¿½crit : > >> Hi FT WG, >> >> I am doing my best to socialize the FT proposal at Intel and gathered a piece of feedback to bring back to the WG. >> >> There was a concern that any time the user registers an error handler, fault tolerance could be "switched on" because MPI_Comm_set_errhandler() does not distinguish between error classes. The assumption was that, when switched on, there would be space/time costs associated with fault tolerance. How does the current proposal determine when fault tolerance should be enabled? >> >> One suggested mechanism was to add a function, MPI_Comm_set_faulthandler() that allows the programmer to distinguish between errors and failures. This would allow the runtime to determine when fault tolerance was desired. I think the way this is handled currently is to rely on the implementation switching on/off fault tolerance when the job is launched. >> >> ~Jim. >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by George Bosilca 16 Jul '13

16 Jul '13

Jim, On Jul 16, 2013, at 22:09 , Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi George, > > Thanks for the detailed response. Let's set aside the MPI_Comm_create_group discussion. You raised a good point about the difficulty in reaching consensus on the group argument that has not come up before. If there is some mechanism for consensus, then Comm_create_group will just work. We can hardly settle here. I was not trying to pinpoint the difficulty of using MPI_Comm_create_group to build a new communicator, I was gently trying to suggest the impossibility of doing so with only the MPI_Comm_create_group call. Using MPI_Comm_agree to agree that the creation of the communicator based on the group (MPI_Comm_create_group) was successful requires you to have a __valid__ (potentially revoked) communicator to do so. You can't use the old communicator, as you will not be able to make a difference between the processes that failed before and after the MPI_Comm_create_group. Similarly, you can't use the newly created communicator either, as it might be incomplete (you did not have an agreement on the group of dead processes on the first place). However, as I mentioned before, there are scenarios where you can use this function if complemented with a shrink (that will in fact build both the consistent knowledge about the dead processes and a communicator to further agree on). > I still have some remaining concerns about supporting a usage model where the communicator has "holes" in it. > > The group query routine that I was thinking of is a local query, which would return information on which processes I know to have failed. Any particular reason the current get_acked() is not satisfying? > An application won't be able to track failures that occur within libraries, Within libraries? > so this query is necessary to find out what is locally known. I assume that MPI will have to track this information anyway, in case you attempt to communicate with a failed process. There is no such requirement from the MPI library in the current proposal. In many applications global knowledge is not required, so there is no reason to propagate such information and force the associated overhead on the application. In these applications the failure will eventually be discovered if a communication with the dead process is initiated. > Consensus. Is it true that right now, that the only ways to reach consensus on the failed group is to revoke the communicator, shrink it, and then compare groups, or to implement your own consensus protocol on top of send/recv? The concern I have is that the revoke method requires the application to destroy the old communicator and renumber the processes, which the programmer may not want to do. The ranks could have some important meaning to the application. As I hinted in my previous email there are many other ways to turn around this problem. You can use topologies, or you can use a shadow communicator that will deal with the revoke operations, while your "working" communicator (which is meant only for point-to-point communications) remains "unrevoked" (but eventually the failures will be acknowledged). > I am probably going to regret asking this, but is it possible to include an MPI_Comm_resume() function that re-activates a revoked communicator with holes in it? For many reasons related to the complexities of distributed systems (lack of synchronization in the error detection, divergent view of the entire system from each process) this operation must have a consensus meaning. Thus in terms of cost it is similar to MPI_Comm_shrink (except the reordering of the processes). It might provide some limited benefit, for people that want to use such type of scenario. Now, if by doing the re-enable you expect that the communicator will behave as a freshly new communicator and everything MPI-related, file, one-sided, collective will just work on this communicator with holes … then we're talking about something so complex that I would not even dare considering for inclusion in the standard. George. > > ~Jim. > > > On Mon, Jul 15, 2013 at 9:28 AM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > Jim, let me try to articulate a comprehensible answer without going too deep into technicalities. > > On Jul 10, 2013, at 16:07 , Jim Dinan <james.dinan(a)gmail.com> wrote: > >> Hi George, >> >>> Re: comm_create and friends -- I'd be interested in hearing the result of the F2F discussion. I've been the person complaining about this, so if it would be helpful for me to join you guys over the phone next Friday, let me know. >> >> I did not have the opportunity to hear your complaint about this. Can you please summarize it here on the mailing list to make sure 1) that we are all at the same level of understanding; 2) the complaint reach all the interested audience and 3) we have the opportunity to address it as accurately as possible. >> >> I'm concerned that the current spec is missing a few features that are needed for the usage model where users continue to use a communicator with "holes" in it. In order for this model to work well, users must be able to query which processes are known to have failed in a given communicator. > > The proposed FT framework was designed to address a wide spectrum of possible approaches, by covering two extreme cases, on one side where global knowledge and global actions are required and on the other side where only local knowledge and local actions are required, while providing building blocks for all other intermediary approaches. > > Looking specifically at your concern it seems that you want to continue to use a communicator with holes while avoiding the revoke, but expect to need global knowledge about the failures. Thus you propose to have a special call GET_ALL_FAILED that provides you the list of failed processes. Interesting idea but unfortunately it has pitfalls. > > - In order to be able to use the list of failed processes to create new communicators using the COMM_CREATE_GROUP this list should be identical on all processes. Therefore the call to GET_ALL_FAILED is similar to an agreement (collective and must have a consensus meaning). Then the obvious question will be how do you synchronize all the living processes to call this blocking function together? This is why you need an atomic broadcast (guaranteed delivery despite failure) that can interrupt the normal behavior of the application. And this is typically the goal of the revoke functionality. You could implement your own protocol of failure knowledge propagation inside your communication scheme, but imagine how much your code will be impacted by this: at any point where you do a communication, you must be able to receive either a normal message of your application, or a notification message. > >>> without the revoke functionality (the knowledge about the fault is only propagated to processes that directly communicate with the dead nodes)? Do you really intend to implement your own protocol of failure knowledge propagation integrated in the application? >>> >>> >>>> Given the current interface, users are not able to query >>>> the set of failed processes without creating a new communicator and >>>> translating ranks, via MPI_Comm_shrink. This requires the user to first >>>> revoke the parent communicator, which is something we want to avoid. >>> >>> Again the revoke is an optional step. If you have a special way to have a consensus over all still alive processes about the list of dead processes, you just have to use it. However, for the sake of performance and portability we strongly believe that MPI should propose such a functionality for the users, that unlike you, don't want to implement their own consensus. >>> >>> >>>> Along these same lines, we should also ensure that MPI_Comm_create_group >>>> will work as expected when a communicator with holes is used, provided >>>> the output group excludes failed processes (i.e., the operation is not >>>> collective over any failed processes). >>> >>> I fail to see your concern here as I can't imagine which of the part of the current proposal prevent you from using such an approach. So from a technical point of view this approach should work in the context of the current proposal >>> >>> Now from a performance point of view the story is slightly different. While in the case of MPI_Comm_shrink highly optimized implementations have the opportunity for merge all the stage of the operation (consensus over the dead processes and creation of a new communicator) in a single step, your approach will have to validate (via an agreement) that the newly created communicator is indeed valid (aka, no new failure has been discovered during the communicator creation). > >> Given the current interface, users are not able to query the set of failed processes without creating a new communicator and translating ranks, via MPI_Comm_shrink. This requires the user to first revoke the parent communicator, which is something we want to avoid. > > Again the revoke is an optional step. If you have a special way to have a consensus over all still alive processes about the list of dead processes, you just have to use it. However, for the sake of performance and portability we strongly believe that MPI should propose such a functionality for the users, that unlike you, don't want to implement their own consensus. > >> Along these same lines, we should also ensure that MPI_Comm_create_group will work as expected when a communicator with holes is used, provided the output group excludes failed processes (i.e., the operation is not collective over any failed processes). > > I fail to see your concern here as I can't imagine which of the part of the current proposal prevent you from using such an approach. So from a technical point of view this approach should work in the context of the current proposal > > Now from a performance point of view the story is slightly different. While in the case of MPI_Comm_shrink highly optimized implementations have the opportunity for merge all the stage of the operation (consensus over the dead processes and creation of a new communicator) in a single step, your approach will have to validate (via an agreement) that the newly created communicator is indeed valid (aka, no new failure has been discovered during the communicator creation). And to do so, you will have to have a communicator without holes, because agree will not provide you appropriate information on communicators with holes. > >> This might not require any text changes, unless we want to allow this operation on revoked communicators. > > At this point I get confused as I was under the impression that your goal was to avoid revoking the communicator? > > Anyway, we do not want original meaning of COMM_CREATE_GROUP to be tainted by FT semantic. Especially true for comm_create_group which is supposed to be super scalable. See above for why it serves little purpose anyway, if it is not to become an expensive agreement operation. > > George. > >>> Re: Roadmap -- Before a reading, it might be helpful to give a brief presentation to the Forum again giving the high level ideas and justifications for each new addition to the FT proposal. I think it's been long enough that people have forgotten the details and this might help them feel more comfortable that the proposal is complete and self-consistent. >> >> There was at least one [more or less] "brief" presentation of the FT proposal at every meeting for the last two years. I would even emphasize the fact that over the last year no major modification of the proposal has been put forward, fact that might indicate a certain level of completeness and self-consistency. >> >> I'm just trying to convey the temperature in the room, as I felt it. I think a 15 minute, very high level warm-up immediately before the reading on the usage models, big ideas, and conventions would go a long way toward prepping the audience. >> >> ~Jim. >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Distinguishing errors from failures
by Aurélien Bouteiller 16 Jul '13

16 Jul '13

Jim, There is no specific cost to enable FT at all time, so this might not be such a crucial issue actually, since the initial assumption is not true. You can point to the following paper that investigates the cost sustained by applications and stressful micro benchmarks to support this claim with solid evidence. Bland, W., Bouteiller, A., Herault, T., Hursey, J., Bosilca, G., Dongarra, J.J. ï¿½An Evaluation of User-Level Failure Mitigation Support in MPI,ï¿½ Computing, Springer, 2013, issn 0010-4885X, http://dx.doi.org/10.1007/s00607-013-0331-3 The intuitive explanation to the no-impact result is that the spec does not change the behavior of any existing MPI functions. If a function has local completion, it retains local completion even when it reports a failure. It merely adds a new class of errors to let users know that what killed them is process failure, rather than say network retransmit error. The FT additions kick in only -after- a failure was reported. Nothing is implicit. The MPI implementation is not expected to "fix" things on its own in the background, normal MPI functions are not overloaded with failure related actives. All recovery actions are triggered by explicit calls from the user code, which removes all potential performance problems from "spilling" recovery activity inside failure-free path. All this means that the codebase (see the prototype implementation) stays basically unchanged, with no modifications in the transport layer, no modifications in the collective framework, etc. ~~~~~~ As you noted, it can also be turned on/off by mpirun switches if deemed necessary. It is standard compliant (and this is deliberate) to have all FT recovery routines map to no-op (revoke=no-op) or no-FT equivalents (agree=allreduce) when no fault tolerance is needed, and even FT codes will run fine on such an MPI library (as long as there are no failures, of course). If some implementation really wants to "optimize out" anything FT related (or even load-in only if required), This would be our recommendation. But again, the performance gain is expected to be minor, if even measurable. Aurelien Le 16 juil. 2013 ï¿½ 16:23, Jim Dinan <james.dinan(a)gmail.com> a ï¿½crit : > Hi FT WG, > > I am doing my best to socialize the FT proposal at Intel and gathered a piece of feedback to bring back to the WG. > > There was a concern that any time the user registers an error handler, fault tolerance could be "switched on" because MPI_Comm_set_errhandler() does not distinguish between error classes. The assumption was that, when switched on, there would be space/time costs associated with fault tolerance. How does the current proposal determine when fault tolerance should be enabled? > > One suggested mechanism was to add a function, MPI_Comm_set_faulthandler() that allows the programmer to distinguish between errors and failures. This would allow the runtime to determine when fault tolerance was desired. I think the way this is handled currently is to rely on the implementation switching on/off fault tolerance when the job is launched. > > ~Jim. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

[Mpi3-ft] Distinguishing errors from failures
by Jim Dinan 16 Jul '13

16 Jul '13

Hi FT WG, I am doing my best to socialize the FT proposal at Intel and gathered a piece of feedback to bring back to the WG. There was a concern that any time the user registers an error handler, fault tolerance could be "switched on" because MPI_Comm_set_errhandler() does not distinguish between error classes. The assumption was that, when switched on, there would be space/time costs associated with fault tolerance. How does the current proposal determine when fault tolerance should be enabled? One suggested mechanism was to add a function, MPI_Comm_set_faulthandler() that allows the programmer to distinguish between errors and failures. This would allow the runtime to determine when fault tolerance was desired. I think the way this is handled currently is to rely on the implementation switching on/off fault tolerance when the job is launched. ~Jim.

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by Jim Dinan 16 Jul '13

16 Jul '13

Hi George, Thanks for the detailed response. Let's set aside the MPI_Comm_create_group discussion. You raised a good point about the difficulty in reaching consensus on the group argument that has not come up before. If there is some mechanism for consensus, then Comm_create_group will just work. I still have some remaining concerns about supporting a usage model where the communicator has "holes" in it. The group query routine that I was thinking of is a local query, which would return information on which processes I know to have failed. An application won't be able to track failures that occur within libraries, so this query is necessary to find out what is locally known. I assume that MPI will have to track this information anyway, in case you attempt to communicate with a failed process. Consensus. Is it true that right now, that the only ways to reach consensus on the failed group is to revoke the communicator, shrink it, and then compare groups, or to implement your own consensus protocol on top of send/recv? The concern I have is that the revoke method requires the application to destroy the old communicator and renumber the processes, which the programmer may not want to do. The ranks could have some important meaning to the application. I am probably going to regret asking this, but is it possible to include an MPI_Comm_resume() function that re-activates a revoked communicator with holes in it? ~Jim. On Mon, Jul 15, 2013 at 9:28 AM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > Jim, let me try to articulate a comprehensible answer without going too > deep into technicalities. > > On Jul 10, 2013, at 16:07 , Jim Dinan <james.dinan(a)gmail.com> wrote: > > Hi George, > > Re: comm_create and friends -- I'd be interested in hearing the result of >> the F2F discussion. I've been the person complaining about this, so if it >> would be helpful for me to join you guys over the phone next Friday, let me >> know. >> >> >> I did not have the opportunity to hear your complaint about this. Can you >> please summarize it here on the mailing list to make sure 1) that we are >> all at the same level of understanding; 2) the complaint reach all the >> interested audience and 3) we have the opportunity to address it as >> accurately as possible. >> > > I'm concerned that the current spec is missing a few features that are > needed for the usage model where users continue to use a communicator with > "holes" in it. In order for this model to work well, users must be able to > query which processes are known to have failed in a given communicator. > > > The proposed FT framework was designed to address a wide spectrum of > possible approaches, by covering two extreme cases, on one side where > global knowledge and global actions are required and on the other side > where only local knowledge and local actions are required, while providing > building blocks for all other intermediary approaches. > > Looking specifically at your concern it seems that you want to continue to > use a communicator with holes while avoiding the revoke, but expect to need > global knowledge about the failures. Thus you propose to have a special > call GET_ALL_FAILED that provides you the list of failed processes. > Interesting idea but unfortunately it has pitfalls. > > - In order to be able to use the list of failed processes to create new > communicators using the COMM_CREATE_GROUP this list should be identical on > all processes. Therefore the call to GET_ALL_FAILED is similar to an > agreement (collective and must have a consensus meaning). Then the obvious > question will be how do you synchronize all the living processes to call > this blocking function together? This is why you need an atomic broadcast > (guaranteed delivery despite failure) that can interrupt the normal > behavior of the application. And this is typically the goal of the revoke > functionality. You could implement your own protocol of failure knowledge > propagation inside your communication scheme, but imagine how much your > code will be impacted by this: at any point where you do a communication, > you must be able to receive either a normal message of your application, or > a notification message. > > without the revoke functionality (the knowledge about the fault is only > propagated to processes that directly communicate with the dead nodes)? Do > you really intend to implement your own protocol of failure knowledge > propagation integrated in the application? > > > Given the current interface, users are not able to query > the set of failed processes without creating a new communicator and > translating ranks, via MPI_Comm_shrink. This requires the user to first > revoke the parent communicator, which is something we want to avoid. > > > Again the revoke is an optional step. If you have a special way to have a > consensus over all still alive processes about the list of dead processes, > you just have to use it. However, for the sake of performance and > portability we strongly believe that MPI should propose such a > functionality for the users, that unlike you, don't want to implement their > own consensus. > > > Along these same lines, we should also ensure that MPI_Comm_create_group > will work as expected when a communicator with holes is used, provided > the output group excludes failed processes (i.e., the operation is not > collective over any failed processes). > > > I fail to see your concern here as I can't imagine which of the part of > the current proposal prevent you from using such an approach. So from a > technical point of view this approach should work in the context of the > current proposal > > Now from a performance point of view the story is slightly different. > While in the case of MPI_Comm_shrink highly optimized implementations have > the opportunity for merge all the stage of the operation (consensus over > the dead processes and creation of a new communicator) in a single step, > your approach will have to validate (via an agreement) that the newly > created communicator is indeed valid (aka, no new failure has been > discovered during the communicator creation). > > > Given the current interface, users are not able to query the set of > failed processes without creating a new communicator and translating ranks, > via MPI_Comm_shrink. This requires the user to first revoke the parent > communicator, which is something we want to avoid. > > > Again the revoke is an optional step. If you have a special way to have a > consensus over all still alive processes about the list of dead processes, > you just have to use it. However, for the sake of performance and > portability we strongly believe that MPI should propose such a > functionality for the users, that unlike you, don't want to implement their > own consensus. > > Along these same lines, we should also ensure that MPI_Comm_create_group > will work as expected when a communicator with holes is used, provided the > output group excludes failed processes (i.e., the operation is not > collective over any failed processes). > > > I fail to see your concern here as I can't imagine which of the part of > the current proposal prevent you from using such an approach. So from a > technical point of view this approach should work in the context of the > current proposal > > Now from a performance point of view the story is slightly different. > While in the case of MPI_Comm_shrink highly optimized implementations have > the opportunity for merge all the stage of the operation (consensus over > the dead processes and creation of a new communicator) in a single step, > your approach will have to validate (via an agreement) that the newly > created communicator is indeed valid (aka, no new failure has been > discovered during the communicator creation). And to do so, you will have > to have a communicator without holes, because agree will not provide you > appropriate information on communicators with holes. > > This might not require any text changes, unless we want to allow this > operation on revoked communicators. > > > At this point I get confused as I was under the impression that your goal > was to avoid revoking the communicator? > > Anyway, we do not want original meaning of COMM_CREATE_GROUP to be tainted > by FT semantic. Especially true for comm_create_group which is supposed to > be super scalable. See above for why it serves little purpose anyway, if it > is not to become an expensive agreement operation. > > George. > > Re: Roadmap -- Before a reading, it might be helpful to give a brief >> presentation to the Forum again giving the high level ideas and >> justifications for each new addition to the FT proposal. I think it's been >> long enough that people have forgotten the details and this might help them >> feel more comfortable that the proposal is complete and self-consistent. >> >> >> There was at least one [more or less] "brief" presentation of the FT >> proposal at every meeting for the last two years. I would even emphasize >> the fact that over the last year no major modification of the proposal has >> been put forward, fact that might indicate a certain level of completeness >> and self-consistency. >> > > I'm just trying to convey the temperature in the room, as I felt it. I > think a 15 minute, very high level warm-up immediately before the reading > on the usage models, big ideas, and conventions would go a long way toward > prepping the audience. > > ~Jim. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0