September 2010 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 22 Sep '10

22 Sep '10

You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 02:41 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org That's why I feel that this is a minor change. As to your example, it's possible that failed collectives result in CANNOT_CONTINUE, but a send to a failed process maybe doesn't have to. You can still send to other processes. -d On Sep 22, 2010, at 1:16 PM, Richard Treumann wrote: > The situation of MPI state being totally trashed by an error that returns a return code barely exists. The case where it is subtly discombobulated is the norm. _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Darius Buntinas 22 Sep '10

22 Sep '10

That's why I feel that this is a minor change. As to your example, it's possible that failed collectives result in CANNOT_CONTINUE, but a send to a failed process maybe doesn't have to. You can still send to other processes. -d On Sep 22, 2010, at 1:16 PM, Richard Treumann wrote: > The situation of MPI state being totally trashed by an error that returns a return code barely exists. The case where it is subtly discombobulated is the norm.

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 22 Sep '10

22 Sep '10

Darius I can imagine a few errors that I know will be harmless to MPI state. I can make sure nobody can do any harm by passing an invalid communicator to MPI_COMM_SIZE. I cannot think of a detectable error that would return and leave that thread of that process so totally broken that nothing in MPI will work from then on. In a collective, there may be processes in which the thread that called the CC never returns and that tread of the process is no longer usable because it is hung. Other threads using other communicators in the process with a hung thread may work perfectly. Except for the very few cases where I know there was no damage (like a bad comm on MPI_COMM_SIZE) the situation, 99.99% of the time, will be that everything still works but sometimes the outcome is a surprise to the user. Say you do: 1 MPI_Barrier (on world) 2 MPI_Barrier (on world): 3 other stuff 4 MPI_Barrier (on world) 5 if (my rank is even) 6 sendrecv(with odd neighbor) 7 else 8 sendrecv(with even neighbor) but get back an error at all even numbered ranks from the line 1 barrier call. The line 2 MPI_Barrier may still "work" but the line 2 barrier at even numbered ranks will match the line 1 barrier at odd ranks. Even ranks will begin "other stuff" and odd ranks will sit in the line 2 barrier until even ranks finish "other stuff" and reach the line 4 barrier. The odd ranks now get through their line 2 barrier and begin other stuff. If "other stuff" involves communication among the even ranks and communication among the odd ranks. that will work too. The even ranks will all send/recv among themselves later the odd ranks will all send/recv among themselves. The even ranks will reach line 6 and hang there because the odd tasks are still stuck at line 4. In this entire example, libmpi has continued working "correctly" but the behavior you get from correct behavior is not what you planned. The situation of MPI state being totally trashed by an error that returns a return code barely exists. The case where it is subtly discombobulated is the norm. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 12:24 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org OK, I (think I) see what you guys are saying, so maybe we should look at it this way. The CANNOT_CONTINUE proposal should not define the operation of the MPI implementation after errors other than CANNOT_CONTINUE. Instead, it defines that after the implementation gives a CANNOT_CONTINUE error, the app knows that the implementation is fatally wedged, and that the user should definately not expect correct operation after this. I.e., we're not labeling other errors as "recoverable," we're just marking CANNOT_CONTINUE as "unrecoverable." Note that an implementation can still be standard compliant even if it never returns a CANNOT_CONTINUE error even when it is fatally wedged (because operation after any other error is still undefined). This just defines a way for the implementation to let the user know that it has given up. So that if the implementation provides best-effort functionality after an error, and the user has "read the disclaimer" and is comfortable with proceeding, this is a way to differentiate between an error as a result of a failure that hosed everything, and one that may allow things to continue. We still would like to define what happens to a bcast after a process in the communicator fails. But we leave that for future proposals. Does this make sense? -d On Sep 22, 2010, at 8:43 AM, Terry Dontje wrote: > Richard Treumann wrote: >> >> This proposal is not a minor change. >> >> Please do not make this hole in the standard and assume you can later add language to standardize everything that comes through the hole. >> >> If the standard is to introduce the notion of a recoverable error it must be as part of a full description of what "recovery" means. >> >> I think is is dangerous and ultimately useless to have implementors mark a failure as "recoverable" when the post error state of the distributed MPI has gone from "fully standards compliant" to "mostly standards compliant, read my user doc read my legal disclaimer, cross your fingers". >> >> See comment below for why I do not think the new hole is needed to allow people to do implementation specific recoverability. >> >> There is not even anything to prevent on implementation from deciding to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and documenting 5 or 5000 enumerated values for "answer" ranging from NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. >> >> IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot imagine how we would promise exactly what will work and what will not but in practice most things will still work as expected. >> > I think I agree with Dick on the above. Another way of putting the disagreement is that Josh's proposal is too general in that not all errorcodes can be completely marked as MPI state is broken or not. When Sun implemented fault tolerant client/server we came up with a new error class that when returned gave the user the understanding that a condition occurred on a communicator that has rendered the communicator useless and one should clean it up before continuing on. The point is there was a concrete understanding of the error and what could be done to recover. As opposed to a general class that say's everything is borked or not which essential doesn't give you much because you'll end up eventually having to define a more specific class of error IMO. > > --td >> >> >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: >> >> > [image removed] >> > >> > Re: [Mpi3-ft] Defining the state of MPI after an error >> > >> > Bronis R. de Supinski >> > >> > to: >> > >> > MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> > >> > 09/21/2010 04:59 PM >> > >> > Sent by: >> > >> > mpi3-ft-bounces(a)lists.mpi-forum.org >> > >> > Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance >> > and Dynamic Process Control working Group" >> > >> > >> > Dick: >> > >> > Re: >> > > The current MPI standard does not say the MPI implementation is totally >> > > broken once there is an error. Saying MPI state is undefined after an >> > > error simply says that the detailed semantic of the MPI standard can no >> > > longer be promised. In other words, after an error you leave behind the >> > > security of a portable standard semantic. You are operating at your own >> > > risk. You do not need to read more than that into it. >> > >> > Perhaps my problem with this position is that I come from the >> > background of language definitions for compilers. When you >> > read "undefined" in the OpenMP specification then you are >> > being told that things are broken and the implementation does >> > need to do anything or even tell you what they actually do (and >> > I believe the same is true for the C and C++ standards). An >> > alternative is "implementation defined", which requires the >> > implementer to document what they actually do. Without that, >> > you cannot even rely on actions with a specific implementation >> > (unless you believe "My tests so far have not failed so I am OK"). >> >> >> When a standard says behavior is "undefined" in some situation, it cannot mean behavior is "broken". It cannot mean the implementor is prohibited from making it still work. It cannot mean the implementor is prohibited from making certain things work and documenting them. Any statement like this in a standard would be definition of behavior and the behavior would no longer be "undefined". >> >> The only thing a standard can logically mean by "undefined" is that the STANDARD no longer mandates the definition. >> >> Bronis says: >> >> > >> > I strongly feel "undefined" should be reserved for situations that >> > mean "your program is irrevocably broken and the implementer does >> > not need to worry about what happens to it after encountering them." >> >> I would say this as: >> >> I strongly feel "undefined" should be reserved for situations that mean "The standard no longer guarantees your program is not irrevocably broken. The implementer is not required by the standard to worry about what happens to it after encountering them. An Implementation is free to provide any "better" behavior that may be of value but users cannot assume another implementation provides similar behavior so cannot assume standards defined portability." >> >> I do not see how the use if the word "undefined" in a standard can be interpreted as a prohibition of any behavior an implementation might offer. >> >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> > > > -- > <Mail Attachment.gif> > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle - Performance Technologies > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Darius Buntinas 22 Sep '10

22 Sep '10

Umm, yeah. Let's say a high quality implementation that cannot tolerate faults. But I agree with your statement. -d On Sep 22, 2010, at 12:59 PM, Terry Dontje wrote: > That's funny, because I was thinking a high quality implementation would never return CANNOT_CONTINUE but more distinct error codes that lets an application to recover. I would think very few errors would actually completely obliterate an MPI library's internal structures. At least the implementations I've seen that's seems to be the case.

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Terry Dontje 22 Sep '10

22 Sep '10

Darius Buntinas wrote: > On Sep 22, 2010, at 12:04 PM, Terry Dontje wrote: > > >> Darius Buntinas wrote: >> Ok I need a clarification here because I feel that I might be misinterpreting something. So is the CANNOT_CONTINUE error class only returned by MPI after a previous error condition has been returned that has caused problems? For example let's say we did an MPI_Bcast that resulted in a return of MPI_ERR_OP and for whatever reason the MPI library is borked. So the next call to MPI would return the CANNOT_CONTINUE error class? >> > > Yes, I believe that's the behavior we talked about at the forum. > > >> So is this an escape hatch for an implementation that does not support any type of fault tolerance to explicitly notify the user they shouldn't proceed any further? I really wonder how many implementations will do such. >> >> > > Well, I wouldn't say it's an escape hatch, since an implementation that doesn't support any fault tolerance needn't ever return CANNOT_CONTINUE. Because we still haven't defined what happens after an error, operation after an error is still undefined, the implementation is still free to do anything including returning MPI_SUCCESS. > > However, a high quality implementation, would return CANNOT_CONTINUE on subsequent MPI calls when it knows that something is borked beyond repair. > > Note that there's still a lot of middle ground between "no errors" and "totally borked". We're giving the implementation an error to return if it finds itself totally borked. This is why I agree with Josh's statement that this proposal is a minor change. > > That's funny, because I was thinking a high quality implementation would never return CANNOT_CONTINUE but more distinct error codes that lets an application to recover. I would think very few errors would actually completely obliterate an MPI library's internal structures. At least the implementations I've seen that's seems to be the case. --td > -d > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Oracle Terry D. Dontje | Principal Software Engineer Developer Tools Engineering | +1.781.442.2631 Oracle * - Performance Technologies* 95 Network Drive, Burlington, MA 01803 Email terry.dontje(a)oracle.com <mailto:[email protected]>

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Darius Buntinas 22 Sep '10

22 Sep '10

On Sep 22, 2010, at 12:04 PM, Terry Dontje wrote: > Darius Buntinas wrote: > Ok I need a clarification here because I feel that I might be misinterpreting something. So is the CANNOT_CONTINUE error class only returned by MPI after a previous error condition has been returned that has caused problems? For example let's say we did an MPI_Bcast that resulted in a return of MPI_ERR_OP and for whatever reason the MPI library is borked. So the next call to MPI would return the CANNOT_CONTINUE error class? Yes, I believe that's the behavior we talked about at the forum. > So is this an escape hatch for an implementation that does not support any type of fault tolerance to explicitly notify the user they shouldn't proceed any further? I really wonder how many implementations will do such. > Well, I wouldn't say it's an escape hatch, since an implementation that doesn't support any fault tolerance needn't ever return CANNOT_CONTINUE. Because we still haven't defined what happens after an error, operation after an error is still undefined, the implementation is still free to do anything including returning MPI_SUCCESS. However, a high quality implementation, would return CANNOT_CONTINUE on subsequent MPI calls when it knows that something is borked beyond repair. Note that there's still a lot of middle ground between "no errors" and "totally borked". We're giving the implementation an error to return if it finds itself totally borked. This is why I agree with Josh's statement that this proposal is a minor change. -d

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Terry Dontje 22 Sep '10

22 Sep '10

Darius Buntinas wrote: > OK, I (think I) see what you guys are saying, so maybe we should look at it this way. The CANNOT_CONTINUE proposal should not define the operation of the MPI implementation after errors other than CANNOT_CONTINUE. Instead, it defines that after the implementation gives a CANNOT_CONTINUE error, the app knows that the implementation is fatally wedged, and that the user should definately not expect correct operation after this. I.e., we're not labeling other errors as "recoverable," we're just marking CANNOT_CONTINUE as "unrecoverable." > > Note that an implementation can still be standard compliant even if it never returns a CANNOT_CONTINUE error even when it is fatally wedged (because operation after any other error is still undefined). > > Ok I need a clarification here because I feel that I might be misinterpreting something. So is the CANNOT_CONTINUE error class only returned by MPI after a previous error condition has been returned that has caused problems? For example let's say we did an MPI_Bcast that resulted in a return of MPI_ERR_OP and for whatever reason the MPI library is borked. So the next call to MPI would return the CANNOT_CONTINUE error class? > This just defines a way for the implementation to let the user know that it has given up. So that if the implementation provides best-effort functionality after an error, and the user has "read the disclaimer" and is comfortable with proceeding, this is a way to differentiate between an error as a result of a failure that hosed everything, and one that may allow things to continue. > > So is this an escape hatch for an implementation that does not support any type of fault tolerance to explicitly notify the user they shouldn't proceed any further? I really wonder how many implementations will do such. --td > We still would like to define what happens to a bcast after a process in the communicator fails. But we leave that for future proposals. > > Does this make sense? > -d > > On Sep 22, 2010, at 8:43 AM, Terry Dontje wrote: > > >> Richard Treumann wrote: >> >>> This proposal is not a minor change. >>> >>> Please do not make this hole in the standard and assume you can later add language to standardize everything that comes through the hole. >>> >>> If the standard is to introduce the notion of a recoverable error it must be as part of a full description of what "recovery" means. >>> >>> I think is is dangerous and ultimately useless to have implementors mark a failure as "recoverable" when the post error state of the distributed MPI has gone from "fully standards compliant" to "mostly standards compliant, read my user doc read my legal disclaimer, cross your fingers". >>> >>> See comment below for why I do not think the new hole is needed to allow people to do implementation specific recoverability. >>> >>> There is not even anything to prevent on implementation from deciding to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and documenting 5 or 5000 enumerated values for "answer" ranging from NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. >>> >>> IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot imagine how we would promise exactly what will work and what will not but in practice most things will still work as expected. >>> >>> >> I think I agree with Dick on the above. Another way of putting the disagreement is that Josh's proposal is too general in that not all errorcodes can be completely marked as MPI state is broken or not. When Sun implemented fault tolerant client/server we came up with a new error class that when returned gave the user the understanding that a condition occurred on a communicator that has rendered the communicator useless and one should clean it up before continuing on. The point is there was a concrete understanding of the error and what could be done to recover. As opposed to a general class that say's everything is borked or not which essential doesn't give you much because you'll end up eventually having to define a more specific class of error IMO. >> >> --td >> >>> >>> Dick Treumann - MPI Team >>> IBM Systems & Technology Group >>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>> Tele (845) 433-7846 Fax (845) 433-8363 >>> >>> >>> mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: >>> >>> >>>> [image removed] >>>> >>>> Re: [Mpi3-ft] Defining the state of MPI after an error >>>> >>>> Bronis R. de Supinski >>>> >>>> to: >>>> >>>> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> >>>> 09/21/2010 04:59 PM >>>> >>>> Sent by: >>>> >>>> mpi3-ft-bounces(a)lists.mpi-forum.org >>>> >>>> Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance >>>> and Dynamic Process Control working Group" >>>> >>>> >>>> Dick: >>>> >>>> Re: >>>> >>>>> The current MPI standard does not say the MPI implementation is totally >>>>> broken once there is an error. Saying MPI state is undefined after an >>>>> error simply says that the detailed semantic of the MPI standard can no >>>>> longer be promised. In other words, after an error you leave behind the >>>>> security of a portable standard semantic. You are operating at your own >>>>> risk. You do not need to read more than that into it. >>>>> >>>> Perhaps my problem with this position is that I come from the >>>> background of language definitions for compilers. When you >>>> read "undefined" in the OpenMP specification then you are >>>> being told that things are broken and the implementation does >>>> need to do anything or even tell you what they actually do (and >>>> I believe the same is true for the C and C++ standards). An >>>> alternative is "implementation defined", which requires the >>>> implementer to document what they actually do. Without that, >>>> you cannot even rely on actions with a specific implementation >>>> (unless you believe "My tests so far have not failed so I am OK"). >>>> >>> When a standard says behavior is "undefined" in some situation, it cannot mean behavior is "broken". It cannot mean the implementor is prohibited from making it still work. It cannot mean the implementor is prohibited from making certain things work and documenting them. Any statement like this in a standard would be definition of behavior and the behavior would no longer be "undefined". >>> >>> The only thing a standard can logically mean by "undefined" is that the STANDARD no longer mandates the definition. >>> >>> Bronis says: >>> >>> >>>> I strongly feel "undefined" should be reserved for situations that >>>> mean "your program is irrevocably broken and the implementer does >>>> not need to worry about what happens to it after encountering them." >>>> >>> I would say this as: >>> >>> I strongly feel "undefined" should be reserved for situations that mean "The standard no longer guarantees your program is not irrevocably broken. The implementer is not required by the standard to worry about what happens to it after encountering them. An Implementation is free to provide any "better" behavior that may be of value but users cannot assume another implementation provides similar behavior so cannot assume standards defined portability." >>> >>> I do not see how the use if the word "undefined" in a standard can be interpreted as a prohibition of any behavior an implementation might offer. >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> >>> >> -- >> <Mail Attachment.gif> >> Terry D. Dontje | Principal Software Engineer >> Developer Tools Engineering | +1.781.442.2631 >> Oracle - Performance Technologies >> 95 Network Drive, Burlington, MA 01803 >> Email terry.dontje(a)oracle.com >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Oracle Terry D. Dontje | Principal Software Engineer Developer Tools Engineering | +1.781.442.2631 Oracle * - Performance Technologies* 95 Network Drive, Burlington, MA 01803 Email terry.dontje(a)oracle.com <mailto:[email protected]>

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Bronis R. de Supinski 22 Sep '10

22 Sep '10

Dick: Re: > This proposal is not a minor change. I am not certain to which proposal you are referring. > Please do not make this hole in the standard and assume you can later > add language to standardize everything that comes through the hole. Again, I am not certain to what you are referring. My opinion is that the current status is a hole in the standard. Leaving it simply as "undefined" means that no one can rely on anything other than "It won't work" and they have no leverage to ask what the actual state is. > If the standard is to introduce the notion of a recoverable error it > must be as part of a full description of what "recovery" means. I have not introduced the notion of a recoverable error in what I suggested. I guess you must be referring to Josh's proposal although it is still unclear. I have merely stated that we should require documentation in some situations. > I think is is dangerous and ultimately useless to have implementors > mark a failure as "recoverable" when the post error state of the > distributed MPI has gone from "fully standards compliant" to "mostly > standards compliant, read my user doc read my legal disclaimer, cross > your fingers". My suggestion is not "mostly standards compliant". If you read it that way then you misconstrue what is accepted practice in a wide range of existing standards. > See comment below for why I do not think the new hole is needed to > allow people to do implementation specific recoverability. Obviously, we disagree. I can only say that my position is based on accepted practice in many standards while yours is based on what you feel should be the case. > There is not even anything to prevent on implementation from deciding > to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and > documenting 5 or 5000 enumerated values for "answer" ranging from > NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. I agree there is not. However, you ware under no obligation to document it. "Implementation defined" implies that obligation. Nothing more. > IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot > imagine how we would promise exactly what will work and what will not > but in practice most things will still work as expected. So, ultimately, your position is that the state is undefined in the relevant situations. I find that not very useful. > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: > >> [image removed] >> >> Re: [Mpi3-ft] Defining the state of MPI after an error >> >> Bronis R. de Supinski >> >> to: >> >> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> >> 09/21/2010 04:59 PM >> >> Sent by: >> >> mpi3-ft-bounces(a)lists.mpi-forum.org >> >> Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance >> and Dynamic Process Control working Group" >> >> >> Dick: >> >> Re: >>> The current MPI standard does not say the MPI implementation is totally >>> broken once there is an error. Saying MPI state is undefined after an >>> error simply says that the detailed semantic of the MPI standard can no >>> longer be promised. In other words, after an error you leave behind the >>> security of a portable standard semantic. You are operating at your own >>> risk. You do not need to read more than that into it. >> >> Perhaps my problem with this position is that I come from the >> background of language definitions for compilers. When you >> read "undefined" in the OpenMP specification then you are >> being told that things are broken and the implementation does >> need to do anything or even tell you what they actually do (and >> I believe the same is true for the C and C++ standards). An >> alternative is "implementation defined", which requires the >> implementer to document what they actually do. Without that, >> you cannot even rely on actions with a specific implementation >> (unless you believe "My tests so far have not failed so I am OK"). > > > When a standard says behavior is "undefined" in some situation, it > cannot mean behavior is "broken". It cannot mean the implementor is > prohibited from making it still work. It cannot mean the implementor is > prohibited from making certain things work and documenting them. Any > statement like this in a standard would be definition of behavior and > the behavior would no longer be "undefined". When a standard says behavior is "undefined", it means that ANY behavior is valid AND the implementer has no obligation to document how their implementation behaves. In practice, this means the user must assume that things are broken. You can argue that you document things differently but that does not change the meaning of the standard. > The only thing a standard can logically mean by "undefined" is that the > STANDARD no longer mandates the definition. I guess we agree here. > Bronis says: > >> >> I strongly feel "undefined" should be reserved for situations that >> mean "your program is irrevocably broken and the implementer does >> not need to worry about what happens to it after encountering them." > > I would say this as: > > I strongly feel "undefined" should be reserved for situations that mean > "The standard no longer guarantees your program is not irrevocably > broken. The implementer is not required by the standard to worry about > what happens to it after encountering them. An Implementation is free to > provide any "better" behavior that may be of value but users cannot > assume another implementation provides similar behavior so cannot assume > standards defined portability." Except you are not obligated to document it. I suppose you think that "I can document it if I like so what's the difference" but it is significant in practice. > I do not see how the use if the word "undefined" in a standard can be > interpreted as a prohibition of any behavior an implementation might > offer. At no point did I state that it prohibits the implementation from offering any behavior. I stated that it has no obligation for documentation and that any sane user assumes that things are broken when they are not even assured that documentation will be available. I base this position on experience. Bronis

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Darius Buntinas 22 Sep '10

22 Sep '10

OK, I (think I) see what you guys are saying, so maybe we should look at it this way. The CANNOT_CONTINUE proposal should not define the operation of the MPI implementation after errors other than CANNOT_CONTINUE. Instead, it defines that after the implementation gives a CANNOT_CONTINUE error, the app knows that the implementation is fatally wedged, and that the user should definately not expect correct operation after this. I.e., we're not labeling other errors as "recoverable," we're just marking CANNOT_CONTINUE as "unrecoverable." Note that an implementation can still be standard compliant even if it never returns a CANNOT_CONTINUE error even when it is fatally wedged (because operation after any other error is still undefined). This just defines a way for the implementation to let the user know that it has given up. So that if the implementation provides best-effort functionality after an error, and the user has "read the disclaimer" and is comfortable with proceeding, this is a way to differentiate between an error as a result of a failure that hosed everything, and one that may allow things to continue. We still would like to define what happens to a bcast after a process in the communicator fails. But we leave that for future proposals. Does this make sense? -d On Sep 22, 2010, at 8:43 AM, Terry Dontje wrote: > Richard Treumann wrote: >> >> This proposal is not a minor change. >> >> Please do not make this hole in the standard and assume you can later add language to standardize everything that comes through the hole. >> >> If the standard is to introduce the notion of a recoverable error it must be as part of a full description of what "recovery" means. >> >> I think is is dangerous and ultimately useless to have implementors mark a failure as "recoverable" when the post error state of the distributed MPI has gone from "fully standards compliant" to "mostly standards compliant, read my user doc read my legal disclaimer, cross your fingers". >> >> See comment below for why I do not think the new hole is needed to allow people to do implementation specific recoverability. >> >> There is not even anything to prevent on implementation from deciding to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and documenting 5 or 5000 enumerated values for "answer" ranging from NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. >> >> IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot imagine how we would promise exactly what will work and what will not but in practice most things will still work as expected. >> > I think I agree with Dick on the above. Another way of putting the disagreement is that Josh's proposal is too general in that not all errorcodes can be completely marked as MPI state is broken or not. When Sun implemented fault tolerant client/server we came up with a new error class that when returned gave the user the understanding that a condition occurred on a communicator that has rendered the communicator useless and one should clean it up before continuing on. The point is there was a concrete understanding of the error and what could be done to recover. As opposed to a general class that say's everything is borked or not which essential doesn't give you much because you'll end up eventually having to define a more specific class of error IMO. > > --td >> >> >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: >> >> > [image removed] >> > >> > Re: [Mpi3-ft] Defining the state of MPI after an error >> > >> > Bronis R. de Supinski >> > >> > to: >> > >> > MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> > >> > 09/21/2010 04:59 PM >> > >> > Sent by: >> > >> > mpi3-ft-bounces(a)lists.mpi-forum.org >> > >> > Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance >> > and Dynamic Process Control working Group" >> > >> > >> > Dick: >> > >> > Re: >> > > The current MPI standard does not say the MPI implementation is totally >> > > broken once there is an error. Saying MPI state is undefined after an >> > > error simply says that the detailed semantic of the MPI standard can no >> > > longer be promised. In other words, after an error you leave behind the >> > > security of a portable standard semantic. You are operating at your own >> > > risk. You do not need to read more than that into it. >> > >> > Perhaps my problem with this position is that I come from the >> > background of language definitions for compilers. When you >> > read "undefined" in the OpenMP specification then you are >> > being told that things are broken and the implementation does >> > need to do anything or even tell you what they actually do (and >> > I believe the same is true for the C and C++ standards). An >> > alternative is "implementation defined", which requires the >> > implementer to document what they actually do. Without that, >> > you cannot even rely on actions with a specific implementation >> > (unless you believe "My tests so far have not failed so I am OK"). >> >> >> When a standard says behavior is "undefined" in some situation, it cannot mean behavior is "broken". It cannot mean the implementor is prohibited from making it still work. It cannot mean the implementor is prohibited from making certain things work and documenting them. Any statement like this in a standard would be definition of behavior and the behavior would no longer be "undefined". >> >> The only thing a standard can logically mean by "undefined" is that the STANDARD no longer mandates the definition. >> >> Bronis says: >> >> > >> > I strongly feel "undefined" should be reserved for situations that >> > mean "your program is irrevocably broken and the implementer does >> > not need to worry about what happens to it after encountering them." >> >> I would say this as: >> >> I strongly feel "undefined" should be reserved for situations that mean "The standard no longer guarantees your program is not irrevocably broken. The implementer is not required by the standard to worry about what happens to it after encountering them. An Implementation is free to provide any "better" behavior that may be of value but users cannot assume another implementation provides similar behavior so cannot assume standards defined portability." >> >> I do not see how the use if the word "undefined" in a standard can be interpreted as a prohibition of any behavior an implementation might offer. >> >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> > > > -- > <Mail Attachment.gif> > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle - Performance Technologies > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Terry Dontje 22 Sep '10

22 Sep '10

Richard Treumann wrote: > > This proposal is not a minor change. > > Please do not make this hole in the standard and assume you can later > add language to standardize everything that comes through the hole. > > If the standard is to introduce the notion of a recoverable error it > must be as part of a full description of what "recovery" means. > > I think is is dangerous and ultimately useless to have implementors > mark a failure as "recoverable" when the post error state of the > distributed MPI has gone from "fully standards compliant" to "mostly > standards compliant, read my user doc read my legal disclaimer, cross > your fingers". > > See comment below for why I do not think the new hole is needed to > allow people to do implementation specific recoverability. > > There is not even anything to prevent on implementation from deciding > to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and > documenting 5 or 5000 enumerated values for "answer" ranging from > NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. > > IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot > imagine how we would promise exactly what will work and what will not > but in practice most things will still work as expected. > I think I agree with Dick on the above. Another way of putting the disagreement is that Josh's proposal is too general in that not all errorcodes can be completely marked as MPI state is broken or not. When Sun implemented fault tolerant client/server we came up with a new error class that when returned gave the user the understanding that a condition occurred on a communicator that has rendered the communicator useless and one should clean it up before continuing on. The point is there was a concrete understanding of the error and what could be done to recover. As opposed to a general class that say's everything is borked or not which essential doesn't give you much because you'll end up eventually having to define a more specific class of error IMO. --td > > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: > > > [image removed] > > > > Re: [Mpi3-ft] Defining the state of MPI after an error > > > > Bronis R. de Supinski > > > > to: > > > > MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > > > > 09/21/2010 04:59 PM > > > > Sent by: > > > > mpi3-ft-bounces(a)lists.mpi-forum.org > > > > Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance > > and Dynamic Process Control working Group" > > > > > > Dick: > > > > Re: > > > The current MPI standard does not say the MPI implementation is > totally > > > broken once there is an error. Saying MPI state is undefined > after an > > > error simply says that the detailed semantic of the MPI standard > can no > > > longer be promised. In other words, after an error you leave > behind the > > > security of a portable standard semantic. You are operating at > your own > > > risk. You do not need to read more than that into it. > > > > Perhaps my problem with this position is that I come from the > > background of language definitions for compilers. When you > > read "undefined" in the OpenMP specification then you are > > being told that things are broken and the implementation does > > need to do anything or even tell you what they actually do (and > > I believe the same is true for the C and C++ standards). An > > alternative is "implementation defined", which requires the > > implementer to document what they actually do. Without that, > > you cannot even rely on actions with a specific implementation > > (unless you believe "My tests so far have not failed so I am OK"). > > > When a standard says behavior is "undefined" in some situation, it > cannot mean behavior is "broken". It cannot mean the implementor is > prohibited from making it still work. It cannot mean the implementor > is prohibited from making certain things work and documenting them. > Any statement like this in a standard would be definition of behavior > and the behavior would no longer be "undefined". > > The only thing a standard can logically mean by "undefined" is that > the STANDARD no longer mandates the definition. > > Bronis says: > > > > > I strongly feel "undefined" should be reserved for situations that > > mean "your program is irrevocably broken and the implementer does > > not need to worry about what happens to it after encountering them." > > I would say this as: > > I strongly feel "undefined" should be reserved for situations that > mean "The standard no longer guarantees your program is not > irrevocably broken. The implementer is not required by the standard to > worry about what happens to it after encountering them. An > Implementation is free to provide any "better" behavior that may be of > value but users cannot assume another implementation provides similar > behavior so cannot assume standards defined portability." > > I do not see how the use if the word "undefined" in a standard can be > interpreted as a prohibition of any behavior an implementation might > offer. > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Oracle Terry D. Dontje | Principal Software Engineer Developer Tools Engineering | +1.781.442.2631 Oracle * - Performance Technologies* 95 Network Drive, Burlington, MA 01803 Email terry.dontje(a)oracle.com <mailto:[email protected]>

1 0