June 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] The state of MPI is undefined
by Josh Hursey 13 Jun '11

13 Jun '11

On Mon, Jun 13, 2011 at 12:36 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > I'm not sure I like the wording: > > ï¿½If the MPI implementation can continue operating after process failure > ï¿½then it must return an appropriate error class (e.g., > ï¿½MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined > ï¿½in Chapter 17. > > I don't think we can say that "if MPI returns error code X" ==> "MPI state is defined," because if some state corrupting error occurs, the state of MPI is undefined and the implementation might return error code X. ï¿½Rather, I think we should use the form "if error FOO occurs" ==> "MPI state will be BAR." ï¿½So for example, we can say "if an unrecoverable communication error occurs, the MPI implementation will return a MPI_ERR_COMMUNICATION for all pending communication operations and blah, blah, blah." For the case where the MPI was corrupted so that it is forced to return a particular error code, I do not know how we can get around the fact that we cannot define the state of MPI (the error forced the MPI implementation to lie to the application). So even in the form "if error FOO occurs" ==> "MPI state will be BAR" the 'will be' is too strong since the corruption forced the return of FOO without the consent of the MPI implementation. But if we say 'might be' then we are back to introducing a proposal that cannot be used by applications - lack of solid semantics. So maybe we should add an advice to users that states that: ------------ It is possible that the state of MPI becomes corrupted in an undetectable manner. This undetectable error might force the MPI implementation to return an error code (or even success) unintentionally. For this pathological case there exist no mechanisms to determine the correct running state. A high-quality implementation will strive to return the correct return code from all MPI operations. ------------ > > A blanket statement we can make about errors, then, would be something like "Unless otherwise noted in the standard, after an error is detected, the state of MPI is undefined." > > Another thought, we may get resistance from the Forum about the cost of checking for faults. ï¿½Maybe we discussed this already, I can't remember. ï¿½Do we want to make FT optional, and say "If the MPI implementation supports FT, then the behavior will be blah"? How about: ------------------ MPI does not provide the user with transparent process recovery upon process failure. Once a process fails, MPI does not guarantee that the job can continue or, if the job can continue, that the process can be recovered. If a process failure occurs, and the MPI implementation is able to correctly continue operating after process failure then the MPI implementation will return an appropriate error class (e.g., MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined in Chapter 17. The MPI implementation documentation will provide information on the possible effect of each supported class of errors. Advice to users: It is possible that the state of MPI becomes corrupted in an undetectable manner. This undetectable error might force the MPI implementation to return an error code (or even success) unintentionally. For this pathological case there exist no mechanisms to determine the correct running state. A high-quality implementation will strive to return the correct return code from all MPI operations. ------------------ I like the idea of keeping the softer language with regard to whether or not it is required, but instead say that if the user sees error code X (in this case MPI_ERR_RANK_FAIL_STOP) then they can expect the behavior outlined in Chapter 17. If they do not see that error, but something else then it is implementation defined what will happen afterwards. And the corner case is described to the users, which is an understandable limitation of any software system. What do folks think about this? -- Josh > > -d > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] The state of MPI is undefined
by Darius Buntinas 13 Jun '11

13 Jun '11

I'm not sure I like the wording: If the MPI implementation can continue operating after process failure then it must return an appropriate error class (e.g., MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined in Chapter 17. I don't think we can say that "if MPI returns error code X" ==> "MPI state is defined," because if some state corrupting error occurs, the state of MPI is undefined and the implementation might return error code X. Rather, I think we should use the form "if error FOO occurs" ==> "MPI state will be BAR." So for example, we can say "if an unrecoverable communication error occurs, the MPI implementation will return a MPI_ERR_COMMUNICATION for all pending communication operations and blah, blah, blah." A blanket statement we can make about errors, then, would be something like "Unless otherwise noted in the standard, after an error is detected, the state of MPI is undefined." Another thought, we may get resistance from the Forum about the cost of checking for faults. Maybe we discussed this already, I can't remember. Do we want to make FT optional, and say "If the MPI implementation supports FT, then the behavior will be blah"? -d

1 0

Re: [Mpi3-ft] The state of MPI is undefined
by Josh Hursey 13 Jun '11

13 Jun '11

I included the text from 8.3 so there would be more context in the email regarding what the standard currently says. I thought the plan was to have the modification/clarification early on then leave the text in 8.3 pretty much as is. If the 8.3 test needs to be adjusted then we can do that. The hope was that having an earlier clarification we could avoid duplication much of the text throughout the document. -- Josh On Mon, Jun 13, 2011 at 11:31 AM, Howard Pritchard <howardp(a)cray.com> wrote: > Hi Josh, > > I think your proposed change should be fine. ï¿½Did you intend to > include a proposed change to 8.3 here as well? ï¿½If so, its > missing. > > Howard > > Josh Hursey wrote: >> Here is a suggested paragraph that should probably go in a modified >> version of the existing section 2.8 wording below. >> >> What do folks think about this? Do we need more, less, or something different? >> >> Thanks, >> Josh >> >> >> Discussion: >> ------------------------------------ >> So MPI talks about a couple high level erroneous behavior: >> ï¿½- Section 2.8 "an erroneous MPI call" meaning an MPI call of bad form >> with regard to arguments and matching rules. >> For example, a sending an integer and receiving a float - since >> datatypes are not used in message matching. >> ï¿½- Section 2.8 "reliable message transmission" MPI must mask any >> instability in the networking stack. >> ï¿½- Section 2.8 "processor failures" - nothing defined (no clarification) >> ï¿½- Section 8.3 "error detected" could be an erroneous MPI call or >> internal error. >> >> We would like to say that if the MPI implementation returns >> MPI_ERR_RANK_FAIL stop then they are required to provide the semantics >> from Chapter 17. At the point they can not provide those semantics >> then they must return some other error. >> >> >> Suggest: >> ------------------------------------ >> MPI provides the user with reliable message transmission. >> A message sent is always received correctly, and the user does not >> need to check for transmission errors, time-outs, or other error >> conditions. >> In other words, MPI does not provide mechanisms for dealing with >> failures in the communication system. >> If the MPI implementation is built on an unreliable underlying >> mechanism, then it is the job of the implementor of the MPI subsystem >> to insulate the user from this unreliability, or to reflect >> unrecoverable errors as failures. >> Whenever possible, such failures will be reflected as errors in the >> relevant communication call. >> [REMOVE: Similarly, MPI itself provides no mechanisms for handling >> processor failures.] >> >> MPI does not provide the user with transparent process recovery upon >> process failure. >> Once a process fails, MPI does not guarantee that the job can continue >> or, if the job can continue, that the process can be recovered. >> If the MPI implementation can continue operating after process failure >> then it must return an appropriate error class (e.g., >> MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined >> in Chapter 17. >> The MPI implementation documentation will provide information on the >> possible effect of each supported class of errors. >> >> >> Section 2.8 (Existing) >> ------------------------------------ >> MPI provides the user with reliable message transmission. >> A message sent is always received correctly, and the user does not >> need to check for transmission errors, time-outs, or other error >> conditions. >> In other words, MPI does not provide mechanisms for dealing with >> failures in the communication system. >> If the MPI implementation is built on an unreliable underlying >> mechanism, then it is the job of the implementor of the MPI subsystem >> to insulate the user from this unreliability, or to reflect >> unrecoverable errors as failures. >> Whenever possible, such failures will be reflected as errors in the >> relevant communication call. >> Similarly, MPI itself provides no mechanisms for handling processor failures. >> >> ... >> >> This document does not specify the state of a computation after an >> erroneous MPI call has occurred. >> The desired behavior is that a relevant error code be returned, and >> the effect of the error be localized to the greatest possible extent. >> E.g., it is highly desirable that an erroneous receive call will not >> cause any part of the receiverï¿½s memory to be overwritten, beyond the >> area specified for receiving the message. >> >> Implementations may go beyond this document in supporting in a >> meaningful manner MPI calls that are defined here to be erroneous. >> For example, MPI specifies strict type matching rules between matching >> send and receive operations: it is erroneous to send a floating point >> variable and receive an integer. >> Implementations may go beyond these type matching rules, and provide >> automatic type conversion in such situations. >> It will be helpful to generate warnings for such non-conforming behavior. >> >> >> Section 8.3 (Existing) >> ------------------------------------ >> After an error is detected, the state of MPI is undefined. >> That is, using a user-defined error handler, or MPI_ERRORS_RETURN, >> does not necessarily allow the user to continue to use MPI after an >> error is detected. >> The purpose of these error handlers is to allow a user to issue >> user-defined error messages and to take actions unrelated to MPI (such >> as flushing I/O buffers) before a program exits. >> An MPI implementation is free to allow MPI to continue after an error >> but is not required to do so. >> >> Advice to implementors. >> A good quality implementation will, to the greatest possible extent, >> circumscribe the impact of an error, so that normal processing can >> continue after an error handler was invoked. >> The implementation documentation will provide information on the >> possible effect of each class of errors. >> (End of advice to implementors.) >> >> >> >> On Wed, Jun 8, 2011 at 2:35 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >>> Per our conversation today, we wanted to have a paragraph clearly >>> defining what the MPI standard means by 'After an error is detected, >>> the state of MPI is undefined.'. Since it is defined for some classes >>> of errors. The paragraph would clarify further references of this >>> nature in the MPI standard. >>> >>> Note that this is slightly different than when the program (code) is >>> erroneous due to misuse of the MPI standard interfaces. >>> >>> A few places in the text to look: >>> ï¿½- Section 2.8: Error Handling - Paragraph 1 and 6 >>> ï¿½- Section 8.3: Error Handling - Paragraphs 6 and 7. >>> ï¿½- Section 13.7: I/O Error Handling - Advice to users >>> >>> If the MPI implementation returns an error of MPI_ERR_RANK_FAIL_STOP >>> then it must provide the semantics defined in Chapter 17. We are not, >>> at this time, defining the semantic behavior of the MPI standard after >>> returning other errors. >>> >>> Any suggestions on possible wording? >>> Something like "The state of the computation after an error has >>> occurred may be undefined. A high-quality implementation will continue >>> afterwards. IF the implementation returns an error and the semantics >>> after the error are defined in the standard (e.g., >>> MPI_ERR_RANK_FAIL_STOP in Chapter 17), then the implementation must >>> provide the specified semantics." >>> >>> Any suggestions on where to put the wording? >>> It was suggested that we change/update paragraphs 6 and 7 in Section >>> 8.3 appropriately. >>> >>> >>> Thoughts, >>> Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >> >> >> > > > -- > Howard Pritchard > Software Engineering > Cray, Inc. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] The state of MPI is undefined
by Howard Pritchard 13 Jun '11

13 Jun '11

Hi Josh, I think your proposed change should be fine. Did you intend to include a proposed change to 8.3 here as well? If so, its missing. Howard Josh Hursey wrote: > Here is a suggested paragraph that should probably go in a modified > version of the existing section 2.8 wording below. > > What do folks think about this? Do we need more, less, or something different? > > Thanks, > Josh > > > Discussion: > ------------------------------------ > So MPI talks about a couple high level erroneous behavior: > - Section 2.8 "an erroneous MPI call" meaning an MPI call of bad form > with regard to arguments and matching rules. > For example, a sending an integer and receiving a float - since > datatypes are not used in message matching. > - Section 2.8 "reliable message transmission" MPI must mask any > instability in the networking stack. > - Section 2.8 "processor failures" - nothing defined (no clarification) > - Section 8.3 "error detected" could be an erroneous MPI call or > internal error. > > We would like to say that if the MPI implementation returns > MPI_ERR_RANK_FAIL stop then they are required to provide the semantics > from Chapter 17. At the point they can not provide those semantics > then they must return some other error. > > > Suggest: > ------------------------------------ > MPI provides the user with reliable message transmission. > A message sent is always received correctly, and the user does not > need to check for transmission errors, time-outs, or other error > conditions. > In other words, MPI does not provide mechanisms for dealing with > failures in the communication system. > If the MPI implementation is built on an unreliable underlying > mechanism, then it is the job of the implementor of the MPI subsystem > to insulate the user from this unreliability, or to reflect > unrecoverable errors as failures. > Whenever possible, such failures will be reflected as errors in the > relevant communication call. > [REMOVE: Similarly, MPI itself provides no mechanisms for handling > processor failures.] > > MPI does not provide the user with transparent process recovery upon > process failure. > Once a process fails, MPI does not guarantee that the job can continue > or, if the job can continue, that the process can be recovered. > If the MPI implementation can continue operating after process failure > then it must return an appropriate error class (e.g., > MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined > in Chapter 17. > The MPI implementation documentation will provide information on the > possible effect of each supported class of errors. > > > Section 2.8 (Existing) > ------------------------------------ > MPI provides the user with reliable message transmission. > A message sent is always received correctly, and the user does not > need to check for transmission errors, time-outs, or other error > conditions. > In other words, MPI does not provide mechanisms for dealing with > failures in the communication system. > If the MPI implementation is built on an unreliable underlying > mechanism, then it is the job of the implementor of the MPI subsystem > to insulate the user from this unreliability, or to reflect > unrecoverable errors as failures. > Whenever possible, such failures will be reflected as errors in the > relevant communication call. > Similarly, MPI itself provides no mechanisms for handling processor failures. > > ... > > This document does not specify the state of a computation after an > erroneous MPI call has occurred. > The desired behavior is that a relevant error code be returned, and > the effect of the error be localized to the greatest possible extent. > E.g., it is highly desirable that an erroneous receive call will not > cause any part of the receiverï¿½s memory to be overwritten, beyond the > area specified for receiving the message. > > Implementations may go beyond this document in supporting in a > meaningful manner MPI calls that are defined here to be erroneous. > For example, MPI specifies strict type matching rules between matching > send and receive operations: it is erroneous to send a floating point > variable and receive an integer. > Implementations may go beyond these type matching rules, and provide > automatic type conversion in such situations. > It will be helpful to generate warnings for such non-conforming behavior. > > > Section 8.3 (Existing) > ------------------------------------ > After an error is detected, the state of MPI is undefined. > That is, using a user-defined error handler, or MPI_ERRORS_RETURN, > does not necessarily allow the user to continue to use MPI after an > error is detected. > The purpose of these error handlers is to allow a user to issue > user-defined error messages and to take actions unrelated to MPI (such > as flushing I/O buffers) before a program exits. > An MPI implementation is free to allow MPI to continue after an error > but is not required to do so. > > Advice to implementors. > A good quality implementation will, to the greatest possible extent, > circumscribe the impact of an error, so that normal processing can > continue after an error handler was invoked. > The implementation documentation will provide information on the > possible effect of each class of errors. > (End of advice to implementors.) > > > > On Wed, Jun 8, 2011 at 2:35 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> Per our conversation today, we wanted to have a paragraph clearly >> defining what the MPI standard means by 'After an error is detected, >> the state of MPI is undefined.'. Since it is defined for some classes >> of errors. The paragraph would clarify further references of this >> nature in the MPI standard. >> >> Note that this is slightly different than when the program (code) is >> erroneous due to misuse of the MPI standard interfaces. >> >> A few places in the text to look: >> - Section 2.8: Error Handling - Paragraph 1 and 6 >> - Section 8.3: Error Handling - Paragraphs 6 and 7. >> - Section 13.7: I/O Error Handling - Advice to users >> >> If the MPI implementation returns an error of MPI_ERR_RANK_FAIL_STOP >> then it must provide the semantics defined in Chapter 17. We are not, >> at this time, defining the semantic behavior of the MPI standard after >> returning other errors. >> >> Any suggestions on possible wording? >> Something like "The state of the computation after an error has >> occurred may be undefined. A high-quality implementation will continue >> afterwards. IF the implementation returns an error and the semantics >> after the error are defined in the standard (e.g., >> MPI_ERR_RANK_FAIL_STOP in Chapter 17), then the implementation must >> provide the specified semantics." >> >> Any suggestions on where to put the wording? >> It was suggested that we change/update paragraphs 6 and 7 in Section >> 8.3 appropriately. >> >> >> Thoughts, >> Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-ft] The state of MPI is undefined
by Josh Hursey 13 Jun '11

13 Jun '11

Here is a suggested paragraph that should probably go in a modified version of the existing section 2.8 wording below. What do folks think about this? Do we need more, less, or something different? Thanks, Josh Discussion: ------------------------------------ So MPI talks about a couple high level erroneous behavior: - Section 2.8 "an erroneous MPI call" meaning an MPI call of bad form with regard to arguments and matching rules. For example, a sending an integer and receiving a float - since datatypes are not used in message matching. - Section 2.8 "reliable message transmission" MPI must mask any instability in the networking stack. - Section 2.8 "processor failures" - nothing defined (no clarification) - Section 8.3 "error detected" could be an erroneous MPI call or internal error. We would like to say that if the MPI implementation returns MPI_ERR_RANK_FAIL stop then they are required to provide the semantics from Chapter 17. At the point they can not provide those semantics then they must return some other error. Suggest: ------------------------------------ MPI provides the user with reliable message transmission. A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, or other error conditions. In other words, MPI does not provide mechanisms for dealing with failures in the communication system. If the MPI implementation is built on an unreliable underlying mechanism, then it is the job of the implementor of the MPI subsystem to insulate the user from this unreliability, or to reflect unrecoverable errors as failures. Whenever possible, such failures will be reflected as errors in the relevant communication call. [REMOVE: Similarly, MPI itself provides no mechanisms for handling processor failures.] MPI does not provide the user with transparent process recovery upon process failure. Once a process fails, MPI does not guarantee that the job can continue or, if the job can continue, that the process can be recovered. If the MPI implementation can continue operating after process failure then it must return an appropriate error class (e.g., MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined in Chapter 17. The MPI implementation documentation will provide information on the possible effect of each supported class of errors. Section 2.8 (Existing) ------------------------------------ MPI provides the user with reliable message transmission. A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, or other error conditions. In other words, MPI does not provide mechanisms for dealing with failures in the communication system. If the MPI implementation is built on an unreliable underlying mechanism, then it is the job of the implementor of the MPI subsystem to insulate the user from this unreliability, or to reflect unrecoverable errors as failures. Whenever possible, such failures will be reflected as errors in the relevant communication call. Similarly, MPI itself provides no mechanisms for handling processor failures. ... This document does not specify the state of a computation after an erroneous MPI call has occurred. The desired behavior is that a relevant error code be returned, and the effect of the error be localized to the greatest possible extent. E.g., it is highly desirable that an erroneous receive call will not cause any part of the receiverï¿½s memory to be overwritten, beyond the area specified for receiving the message. Implementations may go beyond this document in supporting in a meaningful manner MPI calls that are defined here to be erroneous. For example, MPI specifies strict type matching rules between matching send and receive operations: it is erroneous to send a floating point variable and receive an integer. Implementations may go beyond these type matching rules, and provide automatic type conversion in such situations. It will be helpful to generate warnings for such non-conforming behavior. Section 8.3 (Existing) ------------------------------------ After an error is detected, the state of MPI is undefined. That is, using a user-defined error handler, or MPI_ERRORS_RETURN, does not necessarily allow the user to continue to use MPI after an error is detected. The purpose of these error handlers is to allow a user to issue user-defined error messages and to take actions unrelated to MPI (such as flushing I/O buffers) before a program exits. An MPI implementation is free to allow MPI to continue after an error but is not required to do so. Advice to implementors. A good quality implementation will, to the greatest possible extent, circumscribe the impact of an error, so that normal processing can continue after an error handler was invoked. The implementation documentation will provide information on the possible effect of each class of errors. (End of advice to implementors.) On Wed, Jun 8, 2011 at 2:35 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Per our conversation today, we wanted to have a paragraph clearly > defining what the MPI standard means by 'After an error is detected, > the state of MPI is undefined.'. Since it is defined for some classes > of errors. The paragraph would clarify further references of this > nature in the MPI standard. > > Note that this is slightly different than when the program (code) is > erroneous due to misuse of the MPI standard interfaces. > > A few places in the text to look: > ï¿½- Section 2.8: Error Handling - Paragraph 1 and 6 > ï¿½- Section 8.3: Error Handling - Paragraphs 6 and 7. > ï¿½- Section 13.7: I/O Error Handling - Advice to users > > If the MPI implementation returns an error of MPI_ERR_RANK_FAIL_STOP > then it must provide the semantics defined in Chapter 17. We are not, > at this time, defining the semantic behavior of the MPI standard after > returning other errors. > > Any suggestions on possible wording? > Something like "The state of the computation after an error has > occurred may be undefined. A high-quality implementation will continue > afterwards. IF the implementation returns an error and the semantics > after the error are defined in the standard (e.g., > MPI_ERR_RANK_FAIL_STOP in Chapter 17), then the implementation must > provide the specified semantics." > > Any suggestions on where to put the wording? > It was suggested that we change/update paragraphs 6 and 7 in Section > 8.3 appropriately. > > > Thoughts, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] General Network Channel Failure
by Kathryn Mohror 12 Jun '11

12 Jun '11

Hi all, (I sent this earlier, but I don't think it went through because I sent it from the wrong email address. I apologize if you get an extra copy.) I know I haven't participated in this working group yet, so I may be missing some context, but I couldn't resist putting my two cents in! I think that an MPI-centric approach is best. Otherwise, you run the risk of defining a model that doesn't fit with a particular implementation or machine and get shot down when it's brought to the forum. For example, you may remember the PERUSE performance interface that assumed a model of MPI that implementers didn't approve, because it didn't fit their implementation or was difficult/expensive to support. Now, to replace PERUSE, we've got the MPI_T interface which doesn't specify *anything* but appears to be supported by the forum. I agree though that having more specific error information when it's available would be very useful. You might consider taking an approach similar to MPI_T -- allow MPI implementers to define any specific error codes they can/want and then provide an interface for decoding and interpreting the errors. Of course, this approach may not be useful for most applications directly, but I imagine that a fault-tolerant MPI application or a checkpoint/restart library could make use of the information, assuming they could get at it. Kathryn On 6/9/2011 8:20 AM, Howard Pritchard wrote: > Hi Greg, > > I vote for an MPI-centric model. > > I also think that part of the job of MPI is to hide as much > as possible things like 'exhaustion of network resources' > and 'intermittent network failures'. Indeed, the very first > sentence in section 2.8 says "MPI provides the user with > reliable message transmission". > > The only reason the topic came up yesterday was in the > context of the fail-stop model and what types of error > codes might be returned by MPI before the official > verdict was in that a fail-stop had occurred. Several of > us checked what our implementations might do prior to > that, and it could include returning MPI_ERR_OTHER. I > could see how for someone writing a fault tolerant MPI > application, something more useful than this rather ambiguous > error code might be worth defining. > > Howard > > > Bronevetsky, Greg wrote: >> I like the idea of having an abstract model of failures that can approximate changes in system functionality due to failures. However, I think before we go too far with this we should consider the type of model we want to make. One option is to make a system model that has as its basic elements nodes, network links and other hardware components and identifies points in time when stop functioning. The other option is to make it MPI-centric by talking about the status of ranks and point-to-point communication between them as well as communicators and collective communication over them. So in the first type of model we can talk about network resource exhaustion and in the latter we can talk about an intermittent inability to send messages over some or all communicators. >> >> I think that the MPI-centric model is a better option since it talks exclusively about entities that exist in MPI and ignores the physical phenomena that cause a given type of degradation in functionality. >> >> The other question we need to discuss is the types of problems we want to represent. We obviously care about fail-stop failures but we're not talking about resource exhaustion. Do we want to add error classes for transient errors and if so, what about performance slowdowns? >> >> Greg Bronevetsky >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky(a)llnl.gov >> http://greg.bronevetsky.com >> >> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>> Sent: Wednesday, June 08, 2011 11:36 AM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> Subject: [Mpi3-ft] General Network Channel Failure >>> >>> It was mentioned in the conversation today that MPI_ERR_RANK_FAIL_STOP >>> may not be the first error returned by an MPI call. In particular the MPI call >>> may return an error symptomatic of a fail-stop process failure (e.g., network >>> link failed - currently MPI_ERR_OTHER), before eventually diagnosing the >>> event as a process failure. This 'space between' MPI_SUCCESS behavior and >>> MPI_ERR_RANK_FAIL_STOP behavior is not currently defined, and probably >>> should be for the application to cleanly move from set of semantics for one >>> error class to another. >>> >>> The suggestion was to create a new general network error class (e.g., >>> MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - MPI_ERR_COMM is >>> taken) that can be returned when the operation cannot complete due to >>> network issues (which might be later diagnosed as process failure and >>> escalated to the MPI_ERR_RANK_FAIL_STOP semantics). You could also think >>> about this error being used for network resource exhaustion as well >>> (something that Tony mentioned during the last MPI Forum meeting). In >>> which case retrying at a later time or taking some other action before trying >>> again would be useful/expected. >>> >>> There are some issues with matching, and the implications on collective >>> operations. If the network error is sticky/permanent then once the error is >>> returned it will always be returned or escalated to fail-stop process failure (or >>> more generally to a 'higher/more severe/more detailed' error class). A >>> recovery proposal (similar to what we are developing for process failure) >>> would allow the application to 'recover' the channel and continue >>> communicating on it. >>> >>> >>> The feeling was that this should be expanded into a full proposal, separate >>> from the Run-Through Stabilization proposal. So we can continue with the >>> RTS proposal, and bring this forward when it is ready. >>> >>> >>> What to folks think about this idea? >>> >>> -- Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] FT Working Group Teleconf
by George R Carr Jr 09 Jun '11

09 Jun '11

>Would people be available to have another teleconf on June 15 >(Wednesday next week at the same time) to continue talking about the >standard document? > >I'm concerned that with only 1 teleconf left on the schedule that we >will not have enough time to discuss the remainder of the document in >time for it to go out the door on July 4. > >Thanks, >Josh > Not my first choice but I can. I may be 10-15 minutes late for the 9AM CDT/10AM MDT start. George -- George R Carr Jr U.S. Department of Commerce NOAA/OAR/GSD Tel: 303-497-4714 Email: George.Carr(a)noaa.gov

1 0

Re: [Mpi3-ft] FT Working Group Teleconf
by Darius Buntinas 09 Jun '11

09 Jun '11

That works for me. -d On Jun 9, 2011, at 1:41 PM, Josh Hursey wrote: > Would people be available to have another teleconf on June 15 > (Wednesday next week at the same time) to continue talking about the > standard document? > > I'm concerned that with only 1 teleconf left on the schedule that we > will not have enough time to discuss the remainder of the document in > time for it to go out the door on July 4. > > Thanks, > Josh > > On Wed, Jun 8, 2011 at 11:37 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> Just a reminder that we meet in <30 min. >> >> On Tue, Jun 7, 2011 at 2:25 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >>> Reminder that we have a teleconf tomorrow, Wed 6/8/2011, at 12 noon EDT. >>> >>> Date: June 8, 2011 >>> Time: 12 noon EDT/New York >>> Dial-in information: >>> 877-801-8130 >>> Code: 1044056 >>> >>> Agenda: >>> - Run-through stabilization proposal review >>> - https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… >>> - https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… >>> - Piggybacking? (Martin S.) >>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking >>> - Process recovery proposal (Josh H.) >>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery >>> - Open Discussion >>> - Other items? >>> >>> Next Meetings: >>> * June 8, 2011 (Teleconf) >>> * June 22, 2011 (Teleconf) >>> * July 6, 2011 (Teleconf) >>> * July 18, 2011 (MPI Forum Meeting) >>> * Aug. 3, 2011 (Teleconf) >>> * Aug. 17, 2011 (Teleconf) >>> * Aug. 31, 2011 (Teleconf) >>> * Sept. 14, 2011 (Teleconf) >>> * Sept. 22, 2011 (MPI Forum Meeting) >>> >>> Respond to the list if you have any additional topics that you would >>> like to discuss. >>> >>> -- Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] FT Working Group Teleconf
by Howard Pritchard 09 Jun '11

09 Jun '11

Hi Josh, I could make it on the 15th. Howard -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Thursday, June 09, 2011 12:42 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] FT Working Group Teleconf Would people be available to have another teleconf on June 15 (Wednesday next week at the same time) to continue talking about the standard document? I'm concerned that with only 1 teleconf left on the schedule that we will not have enough time to discuss the remainder of the document in time for it to go out the door on July 4. Thanks, Josh On Wed, Jun 8, 2011 at 11:37 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Just a reminder that we meet in <30 min. > > On Tue, Jun 7, 2011 at 2:25 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> Reminder that we have a teleconf tomorrow, Wed 6/8/2011, at 12 noon EDT. >> >> Date: June 8, 2011 >> Time: 12 noon EDT/New York >> Dial-in information: >> 877-801-8130 >> Code: 1044056 >> >> Agenda: >> - Run-through stabilization proposal review >> ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… >> ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… >> - Piggybacking? (Martin S.) >> ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking >> - Process recovery proposal (Josh H.) >> ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery >> ï¿½- Open Discussion >> - Other items? >> >> Next Meetings: >> * June ï¿½ 8, 2011 (Teleconf) >> * June ï¿½22, 2011 (Teleconf) >> * July ï¿½ 6, 2011 (Teleconf) >> * July ï¿½18, 2011 (MPI Forum Meeting) >> * Aug. ï¿½ 3, 2011 (Teleconf) >> * Aug. ï¿½17, 2011 (Teleconf) >> * Aug. ï¿½31, 2011 (Teleconf) >> * Sept. 14, 2011 (Teleconf) >> * Sept. 22, 2011 (MPI Forum Meeting) >> >> Respond to the list if you have any additional topics that you would >> like to discuss. >> >> -- Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] FT Working Group Teleconf
by Josh Hursey 09 Jun '11

09 Jun '11

Would people be available to have another teleconf on June 15 (Wednesday next week at the same time) to continue talking about the standard document? I'm concerned that with only 1 teleconf left on the schedule that we will not have enough time to discuss the remainder of the document in time for it to go out the door on July 4. Thanks, Josh On Wed, Jun 8, 2011 at 11:37 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Just a reminder that we meet in <30 min. > > On Tue, Jun 7, 2011 at 2:25 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> Reminder that we have a teleconf tomorrow, Wed 6/8/2011, at 12 noon EDT. >> >> Date: June 8, 2011 >> Time: 12 noon EDT/New York >> Dial-in information: >> 877-801-8130 >> Code: 1044056 >> >> Agenda: >> - Run-through stabilization proposal review >> ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… >> ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… >> - Piggybacking? (Martin S.) >> ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking >> - Process recovery proposal (Josh H.) >> ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery >> ï¿½- Open Discussion >> - Other items? >> >> Next Meetings: >> * June ï¿½ 8, 2011 (Teleconf) >> * June ï¿½22, 2011 (Teleconf) >> * July ï¿½ 6, 2011 (Teleconf) >> * July ï¿½18, 2011 (MPI Forum Meeting) >> * Aug. ï¿½ 3, 2011 (Teleconf) >> * Aug. ï¿½17, 2011 (Teleconf) >> * Aug. ï¿½31, 2011 (Teleconf) >> * Sept. 14, 2011 (Teleconf) >> * Sept. 22, 2011 (MPI Forum Meeting) >> >> Respond to the list if you have any additional topics that you would >> like to discuss. >> >> -- Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0