June 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] General Network Channel Failure
by Howard Pritchard 09 Jun '11

09 Jun '11

Hi Greg, I vote for an MPI-centric model. I also think that part of the job of MPI is to hide as much as possible things like 'exhaustion of network resources' and 'intermittent network failures'. Indeed, the very first sentence in section 2.8 says "MPI provides the user with reliable message transmission". The only reason the topic came up yesterday was in the context of the fail-stop model and what types of error codes might be returned by MPI before the official verdict was in that a fail-stop had occurred. Several of us checked what our implementations might do prior to that, and it could include returning MPI_ERR_OTHER. I could see how for someone writing a fault tolerant MPI application, something more useful than this rather ambiguous error code might be worth defining. Howard Bronevetsky, Greg wrote: > I like the idea of having an abstract model of failures that can approximate changes in system functionality due to failures. However, I think before we go too far with this we should consider the type of model we want to make. One option is to make a system model that has as its basic elements nodes, network links and other hardware components and identifies points in time when stop functioning. The other option is to make it MPI-centric by talking about the status of ranks and point-to-point communication between them as well as communicators and collective communication over them. So in the first type of model we can talk about network resource exhaustion and in the latter we can talk about an intermittent inability to send messages over some or all communicators. > > I think that the MPI-centric model is a better option since it talks exclusively about entities that exist in MPI and ignores the physical phenomena that cause a given type of degradation in functionality. > > The other question we need to discuss is the types of problems we want to represent. We obviously care about fail-stop failures but we're not talking about resource exhaustion. Do we want to add error classes for transient errors and if so, what about performance slowdowns? > > Greg Bronevetsky > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky(a)llnl.gov > http://greg.bronevetsky.com > > >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >> Sent: Wednesday, June 08, 2011 11:36 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: [Mpi3-ft] General Network Channel Failure >> >> It was mentioned in the conversation today that MPI_ERR_RANK_FAIL_STOP >> may not be the first error returned by an MPI call. In particular the MPI call >> may return an error symptomatic of a fail-stop process failure (e.g., network >> link failed - currently MPI_ERR_OTHER), before eventually diagnosing the >> event as a process failure. This 'space between' MPI_SUCCESS behavior and >> MPI_ERR_RANK_FAIL_STOP behavior is not currently defined, and probably >> should be for the application to cleanly move from set of semantics for one >> error class to another. >> >> The suggestion was to create a new general network error class (e.g., >> MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - MPI_ERR_COMM is >> taken) that can be returned when the operation cannot complete due to >> network issues (which might be later diagnosed as process failure and >> escalated to the MPI_ERR_RANK_FAIL_STOP semantics). You could also think >> about this error being used for network resource exhaustion as well >> (something that Tony mentioned during the last MPI Forum meeting). In >> which case retrying at a later time or taking some other action before trying >> again would be useful/expected. >> >> There are some issues with matching, and the implications on collective >> operations. If the network error is sticky/permanent then once the error is >> returned it will always be returned or escalated to fail-stop process failure (or >> more generally to a 'higher/more severe/more detailed' error class). A >> recovery proposal (similar to what we are developing for process failure) >> would allow the application to 'recover' the channel and continue >> communicating on it. >> >> >> The feeling was that this should be expanded into a full proposal, separate >> from the Run-Through Stabilization proposal. So we can continue with the >> RTS proposal, and bring this forward when it is ready. >> >> >> What to folks think about this idea? >> >> -- Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-ft] General Network Channel Failure
by Bronevetsky, Greg 09 Jun '11

09 Jun '11

I like the idea of having an abstract model of failures that can approximate changes in system functionality due to failures. However, I think before we go too far with this we should consider the type of model we want to make. One option is to make a system model that has as its basic elements nodes, network links and other hardware components and identifies points in time when stop functioning. The other option is to make it MPI-centric by talking about the status of ranks and point-to-point communication between them as well as communicators and collective communication over them. So in the first type of model we can talk about network resource exhaustion and in the latter we can talk about an intermittent inability to send messages over some or all communicators. I think that the MPI-centric model is a better option since it talks exclusively about entities that exist in MPI and ignores the physical phenomena that cause a given type of degradation in functionality. The other question we need to discuss is the types of problems we want to represent. We obviously care about fail-stop failures but we're not talking about resource exhaustion. Do we want to add error classes for transient errors and if so, what about performance slowdowns? Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey > Sent: Wednesday, June 08, 2011 11:36 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] General Network Channel Failure > > It was mentioned in the conversation today that MPI_ERR_RANK_FAIL_STOP > may not be the first error returned by an MPI call. In particular the MPI call > may return an error symptomatic of a fail-stop process failure (e.g., network > link failed - currently MPI_ERR_OTHER), before eventually diagnosing the > event as a process failure. This 'space between' MPI_SUCCESS behavior and > MPI_ERR_RANK_FAIL_STOP behavior is not currently defined, and probably > should be for the application to cleanly move from set of semantics for one > error class to another. > > The suggestion was to create a new general network error class (e.g., > MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - MPI_ERR_COMM is > taken) that can be returned when the operation cannot complete due to > network issues (which might be later diagnosed as process failure and > escalated to the MPI_ERR_RANK_FAIL_STOP semantics). You could also think > about this error being used for network resource exhaustion as well > (something that Tony mentioned during the last MPI Forum meeting). In > which case retrying at a later time or taking some other action before trying > again would be useful/expected. > > There are some issues with matching, and the implications on collective > operations. If the network error is sticky/permanent then once the error is > returned it will always be returned or escalated to fail-stop process failure (or > more generally to a 'higher/more severe/more detailed' error class). A > recovery proposal (similar to what we are developing for process failure) > would allow the application to 'recover' the channel and continue > communicating on it. > > > The feeling was that this should be expanded into a full proposal, separate > from the Run-Through Stabilization proposal. So we can continue with the > RTS proposal, and bring this forward when it is ready. > > > What to folks think about this idea? > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] General Network Channel Failure
by Josh Hursey 08 Jun '11

08 Jun '11

It was mentioned in the conversation today that MPI_ERR_RANK_FAIL_STOP may not be the first error returned by an MPI call. In particular the MPI call may return an error symptomatic of a fail-stop process failure (e.g., network link failed - currently MPI_ERR_OTHER), before eventually diagnosing the event as a process failure. This 'space between' MPI_SUCCESS behavior and MPI_ERR_RANK_FAIL_STOP behavior is not currently defined, and probably should be for the application to cleanly move from set of semantics for one error class to another. The suggestion was to create a new general network error class (e.g., MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - MPI_ERR_COMM is taken) that can be returned when the operation cannot complete due to network issues (which might be later diagnosed as process failure and escalated to the MPI_ERR_RANK_FAIL_STOP semantics). You could also think about this error being used for network resource exhaustion as well (something that Tony mentioned during the last MPI Forum meeting). In which case retrying at a later time or taking some other action before trying again would be useful/expected. There are some issues with matching, and the implications on collective operations. If the network error is sticky/permanent then once the error is returned it will always be returned or escalated to fail-stop process failure (or more generally to a 'higher/more severe/more detailed' error class). A recovery proposal (similar to what we are developing for process failure) would allow the application to 'recover' the channel and continue communicating on it. The feeling was that this should be expanded into a full proposal, separate from the Run-Through Stabilization proposal. So we can continue with the RTS proposal, and bring this forward when it is ready. What to folks think about this idea? -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] The state of MPI is undefined
by Josh Hursey 08 Jun '11

08 Jun '11

Per our conversation today, we wanted to have a paragraph clearly defining what the MPI standard means by 'After an error is detected, the state of MPI is undefined.'. Since it is defined for some classes of errors. The paragraph would clarify further references of this nature in the MPI standard. Note that this is slightly different than when the program (code) is erroneous due to misuse of the MPI standard interfaces. A few places in the text to look: - Section 2.8: Error Handling - Paragraph 1 and 6 - Section 8.3: Error Handling - Paragraphs 6 and 7. - Section 13.7: I/O Error Handling - Advice to users If the MPI implementation returns an error of MPI_ERR_RANK_FAIL_STOP then it must provide the semantics defined in Chapter 17. We are not, at this time, defining the semantic behavior of the MPI standard after returning other errors. Any suggestions on possible wording? Something like "The state of the computation after an error has occurred may be undefined. A high-quality implementation will continue afterwards. IF the implementation returns an error and the semantics after the error are defined in the standard (e.g., MPI_ERR_RANK_FAIL_STOP in Chapter 17), then the implementation must provide the specified semantics." Any suggestions on where to put the wording? It was suggested that we change/update paragraphs 6 and 7 in Section 8.3 appropriately. Thoughts, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] FT Working Group Teleconf
by Josh Hursey 08 Jun '11

08 Jun '11

Just a reminder that we meet in <30 min. On Tue, Jun 7, 2011 at 2:25 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Reminder that we have a teleconf tomorrow, Wed 6/8/2011, at 12 noon EDT. > > Date: June 8, 2011 > Time: 12 noon EDT/New York > Dial-in information: > 877-801-8130 > Code: 1044056 > > Agenda: > - Run-through stabilization proposal review > ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… > - Piggybacking? (Martin S.) > ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - Process recovery proposal (Josh H.) > ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery > ï¿½- Open Discussion > - Other items? > > Next Meetings: > * June ï¿½ 8, 2011 (Teleconf) > * June ï¿½22, 2011 (Teleconf) > * July ï¿½ 6, 2011 (Teleconf) > * July ï¿½18, 2011 (MPI Forum Meeting) > * Aug. ï¿½ 3, 2011 (Teleconf) > * Aug. ï¿½17, 2011 (Teleconf) > * Aug. ï¿½31, 2011 (Teleconf) > * Sept. 14, 2011 (Teleconf) > * Sept. 22, 2011 (MPI Forum Meeting) > > Respond to the list if you have any additional topics that you would > like to discuss. > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT Working Group Teleconf
by Josh Hursey 07 Jun '11

07 Jun '11

Reminder that we have a teleconf tomorrow, Wed 6/8/2011, at 12 noon EDT. Date: June 8, 2011 Time: 12 noon EDT/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Run-through stabilization proposal review - https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… - Piggybacking? (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Process recovery proposal (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery - Open Discussion - Other items? Next Meetings: * June 8, 2011 (Teleconf) * June 22, 2011 (Teleconf) * July 6, 2011 (Teleconf) * July 18, 2011 (MPI Forum Meeting) * Aug. 3, 2011 (Teleconf) * Aug. 17, 2011 (Teleconf) * Aug. 31, 2011 (Teleconf) * Sept. 14, 2011 (Teleconf) * Sept. 22, 2011 (MPI Forum Meeting) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] First draft of the Run-Through Stabilization Standard PDF
by Josh Hursey 06 Jun '11

06 Jun '11

I just uploaded to the wiki a first draft of the Run-Through Stabilization proposal in MPI Standard document format. This is the full MPI Standard with the proposal integrated. Most of the fault tolerance language is in Chapter 17. To find the few other changes search the PDF for "ticket276": * https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… * https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… The PDF contains highlighting of all of the 3.0 changes (not just the FT changes). Unfortunately, the latex setup does not allow me to just highlight the changes from our proposal. Please try to review the changes to the document by the teleconf on Wed. I want to talk through any problems that folks find then. You can send notes and comments to the list for further discussion as well. Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0