May 2015 - mpiwg-ft - mailman3.cels.anl.gov

Ticket 324 Updates
by Bland, Wesley 27 May '15

27 May '15

Hi all, Aurelien and I have made some minor changes to ticket 324. There’s a new advice to implementors in the MPI_ABORT definition and the extra phrase “in this situation” in the description of handling a generic fatal error that was discussed on the call yesterday. A new PDF is available on the ticket itself and the slides have been updated. There have been a couple more updates to the slides to make things more clear and better focus the motivation of the ticket. Feel free to make another pass on both and let me know if you think something else needs to be fixed. Thanks, Wesley

1 0

FTWG Con Call TODAY - 2015-05-26
by Bland, Wesley 26 May '15

26 May '15

Today is our usual FTWG con call: Dial-in information: Phone: 712-432-0360 Code: 623998# On the agenda: * Errors are fatal * Ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 * Slides: https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… Still in progress: * Redesign error handlers * RMA ULFM text changes * ULFM Thanks, Wesley

1 0

Re: [mpiwg-ft] Ticket #324 Reading Slides
by George Bosilca 22 May '15

22 May '15

If one takes in account connection transitivity as defined by MPI, it is extremely difficult to implement what you propose as you will obtain a cascade of aborts. Moreover, let's assume that you do set the MPI_ERRORS_RETURN on a communicator, and you notice a failure. What is the state of the other communicators? How do you know that you can use another communicator do continue working? However, if what you want is define the scope of MPI_ABORT in a way that does not prevent implementation specific FT approaches, then why not going for the simplest approach where we clarify MPI_ABORT as aborting at least all processes on the specified communicator, but without the extra modifications that supposedly clarifies the scope of a fatal error? In other words slide 7 without all the rest. Thanks, George. On Thu, May 21, 2015 at 2:26 PM, Bland, Wesley <wesley.bland(a)intel.com> wrote: > I’ve just modified the slides to make this argument a little more obvious. > Let me know if you think it’s not enough. > > On May 21, 2015, at 1:07 PM, Bland, Wesley <wesley.bland(a)intel.com<mailto: > wesley.bland(a)intel.com>> wrote: > > The abort case is pretty simple. I’ve implemented it in MPICH with just an > hour or so’s work. If a process calls abort on a communicator, it sends out > a message to the other processes in its communicator first to tell them to > abort as well. This can also be done by the process manager if it supports > it. At the moment, PMI only handles full job aborts, but that’s something > that might come in PMI-3. > > The failure case is as implementation specific as it always was. If the > implementation detects a failure, it’s supposed to raise an error on the > affected communicators. Those communicators raise their error handlers > (which may include aborting as above). If the implementation can’t/doesn’t > want to do that, it can still just abort everyone like before. > > The use case that this enables in my mind is a more dynamic processing > model where processes are spawned/joined/connected to do a specific task, > then shut down while the rest of the application continues. There’s lots of > communicators between the master(s) and workers and fine grained failure > detection isn’t really necessary because you will just re-spawn a new set > of workers and it’s up to the launcher to avoid failures. For the > inter-communicators, you’d set MPI_ERRORS_RETURN and when you see an error, > you just drop the communicator. The remote processes would have their own > communicator that they could use to abort if necessary. > > It still doesn’t define whether or not communication is possible. That’s a > much bigger issue (obviously) and one that we’re trying to tackle with > ULFM, but for this sort of very simple fault tolerance, the implementation > may be able to continue communicating without an issue as long as you are > going to be destroying all of the communicators with the failed process > anyway. You wouldn’t have to worry about “fixing” MPI_COMM_WORLD since it > wouldn’t have a failure. > > If you want to think about it as part of the overall timeline of improving > FT in the MPI Standard, this really is just a step 0 to ensure that > MPI_Abort works in the correct way. It doesn’t solve everything, but it > does fix a small thing that easy to carve off and doesn’t impact other > parts of the standard too much. > > Thanks, > Wesley > > On May 20, 2015, at 5:53 PM, George Bosilca <bosilca(a)icl.utk.edu<mailto: > bosilca(a)icl.utk.edu>> wrote: > > Wesley, > > I understand the interest in scoping the abort to a single set of > processes. However, without support for detecting that some processes have > disappeared (failure or abort) I don't see how you can use this for > anything constructive. Thus, I am confused by this proposal as it seems to > lack a second part where the processes outside of the MPI_ABORT scope deal > with the newly defined behavior. > > Can you be a little bit more explicit on how this scoped MPI_ABORT, as > defined in this ticket, will be beneficial to application? > > Thanks, > George. > > > > On Tue, May 19, 2015 at 12:14 PM, Bland, Wesley <wesley.bland(a)intel.com > <mailto:[email protected]>> wrote: > I plan to go over this during the con call next week, but if you’d like to > get a preview and/or comment early, I have a draft of the slides for the > reading for ticket #324. You can view the slides here: > > > https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… > > If you have any questions/comments, let me know. > > Aurelien, I’m not sure of the status of this ticket in Open MPI. Is there > a way to keep Open MPI from aborting on any error in the trunk? Is the only > way to get this in the ULFM repo? > > Thanks, > Wesley > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

Re: [mpiwg-ft] Ticket #324 Reading Slides
by Bland, Wesley 21 May '15

21 May '15

I’ve just modified the slides to make this argument a little more obvious. Let me know if you think it’s not enough. On May 21, 2015, at 1:07 PM, Bland, Wesley <wesley.bland(a)intel.com<mailto:[email protected]>> wrote: The abort case is pretty simple. I’ve implemented it in MPICH with just an hour or so’s work. If a process calls abort on a communicator, it sends out a message to the other processes in its communicator first to tell them to abort as well. This can also be done by the process manager if it supports it. At the moment, PMI only handles full job aborts, but that’s something that might come in PMI-3. The failure case is as implementation specific as it always was. If the implementation detects a failure, it’s supposed to raise an error on the affected communicators. Those communicators raise their error handlers (which may include aborting as above). If the implementation can’t/doesn’t want to do that, it can still just abort everyone like before. The use case that this enables in my mind is a more dynamic processing model where processes are spawned/joined/connected to do a specific task, then shut down while the rest of the application continues. There’s lots of communicators between the master(s) and workers and fine grained failure detection isn’t really necessary because you will just re-spawn a new set of workers and it’s up to the launcher to avoid failures. For the inter-communicators, you’d set MPI_ERRORS_RETURN and when you see an error, you just drop the communicator. The remote processes would have their own communicator that they could use to abort if necessary. It still doesn’t define whether or not communication is possible. That’s a much bigger issue (obviously) and one that we’re trying to tackle with ULFM, but for this sort of very simple fault tolerance, the implementation may be able to continue communicating without an issue as long as you are going to be destroying all of the communicators with the failed process anyway. You wouldn’t have to worry about “fixing” MPI_COMM_WORLD since it wouldn’t have a failure. If you want to think about it as part of the overall timeline of improving FT in the MPI Standard, this really is just a step 0 to ensure that MPI_Abort works in the correct way. It doesn’t solve everything, but it does fix a small thing that easy to carve off and doesn’t impact other parts of the standard too much. Thanks, Wesley On May 20, 2015, at 5:53 PM, George Bosilca <bosilca(a)icl.utk.edu<mailto:[email protected]>> wrote: Wesley, I understand the interest in scoping the abort to a single set of processes. However, without support for detecting that some processes have disappeared (failure or abort) I don't see how you can use this for anything constructive. Thus, I am confused by this proposal as it seems to lack a second part where the processes outside of the MPI_ABORT scope deal with the newly defined behavior. Can you be a little bit more explicit on how this scoped MPI_ABORT, as defined in this ticket, will be beneficial to application? Thanks, George. On Tue, May 19, 2015 at 12:14 PM, Bland, Wesley <wesley.bland(a)intel.com<mailto:[email protected]>> wrote: I plan to go over this during the con call next week, but if you’d like to get a preview and/or comment early, I have a draft of the slides for the reading for ticket #324. You can view the slides here: https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… If you have any questions/comments, let me know. Aurelien, I’m not sure of the status of this ticket in Open MPI. Is there a way to keep Open MPI from aborting on any error in the trunk? Is the only way to get this in the ULFM repo? Thanks, Wesley _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Ticket #324 Reading Slides
by Bland, Wesley 21 May '15

21 May '15

The abort case is pretty simple. I’ve implemented it in MPICH with just an hour or so’s work. If a process calls abort on a communicator, it sends out a message to the other processes in its communicator first to tell them to abort as well. This can also be done by the process manager if it supports it. At the moment, PMI only handles full job aborts, but that’s something that might come in PMI-3. The failure case is as implementation specific as it always was. If the implementation detects a failure, it’s supposed to raise an error on the affected communicators. Those communicators raise their error handlers (which may include aborting as above). If the implementation can’t/doesn’t want to do that, it can still just abort everyone like before. The use case that this enables in my mind is a more dynamic processing model where processes are spawned/joined/connected to do a specific task, then shut down while the rest of the application continues. There’s lots of communicators between the master(s) and workers and fine grained failure detection isn’t really necessary because you will just re-spawn a new set of workers and it’s up to the launcher to avoid failures. For the inter-communicators, you’d set MPI_ERRORS_RETURN and when you see an error, you just drop the communicator. The remote processes would have their own communicator that they could use to abort if necessary. It still doesn’t define whether or not communication is possible. That’s a much bigger issue (obviously) and one that we’re trying to tackle with ULFM, but for this sort of very simple fault tolerance, the implementation may be able to continue communicating without an issue as long as you are going to be destroying all of the communicators with the failed process anyway. You wouldn’t have to worry about “fixing” MPI_COMM_WORLD since it wouldn’t have a failure. If you want to think about it as part of the overall timeline of improving FT in the MPI Standard, this really is just a step 0 to ensure that MPI_Abort works in the correct way. It doesn’t solve everything, but it does fix a small thing that easy to carve off and doesn’t impact other parts of the standard too much. Thanks, Wesley On May 20, 2015, at 5:53 PM, George Bosilca <bosilca(a)icl.utk.edu<mailto:[email protected]>> wrote: Wesley, I understand the interest in scoping the abort to a single set of processes. However, without support for detecting that some processes have disappeared (failure or abort) I don't see how you can use this for anything constructive. Thus, I am confused by this proposal as it seems to lack a second part where the processes outside of the MPI_ABORT scope deal with the newly defined behavior. Can you be a little bit more explicit on how this scoped MPI_ABORT, as defined in this ticket, will be beneficial to application? Thanks, George. On Tue, May 19, 2015 at 12:14 PM, Bland, Wesley <wesley.bland(a)intel.com<mailto:[email protected]>> wrote: I plan to go over this during the con call next week, but if you’d like to get a preview and/or comment early, I have a draft of the slides for the reading for ticket #324. You can view the slides here: https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… If you have any questions/comments, let me know. Aurelien, I’m not sure of the status of this ticket in Open MPI. Is there a way to keep Open MPI from aborting on any error in the trunk? Is the only way to get this in the ULFM repo? Thanks, Wesley _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Ticket #324 Reading Slides
by George Bosilca 20 May '15

20 May '15

Wesley, I understand the interest in scoping the abort to a single set of processes. However, without support for detecting that some processes have disappeared (failure or abort) I don't see how you can use this for anything constructive. Thus, I am confused by this proposal as it seems to lack a second part where the processes outside of the MPI_ABORT scope deal with the newly defined behavior. Can you be a little bit more explicit on how this scoped MPI_ABORT, as defined in this ticket, will be beneficial to application? Thanks, George. On Tue, May 19, 2015 at 12:14 PM, Bland, Wesley <wesley.bland(a)intel.com> wrote: > I plan to go over this during the con call next week, but if you’d like to > get a preview and/or comment early, I have a draft of the slides for the > reading for ticket #324. You can view the slides here: > > > https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… > > If you have any questions/comments, let me know. > > Aurelien, I’m not sure of the status of this ticket in Open MPI. Is there > a way to keep Open MPI from aborting on any error in the trunk? Is the only > way to get this in the ULFM repo? > > Thanks, > Wesley > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Ticket #324 Reading Slides
by Bland, Wesley 19 May '15

19 May '15

I plan to go over this during the con call next week, but if you’d like to get a preview and/or comment early, I have a draft of the slides for the reading for ticket #324. You can view the slides here: https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… If you have any questions/comments, let me know. Aurelien, I’m not sure of the status of this ticket in Open MPI. Is there a way to keep Open MPI from aborting on any error in the trunk? Is the only way to get this in the ULFM repo? Thanks, Wesley

1 0