September 2010 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Defining the state of MPI after an error
by Graham, Richard L. 23 Sep '10

23 Sep '10

Thanks for the comments - responses in line On 9/23/10 5:45 AM, "Terry Dontje" <terry.dontje(a)oracle.com> wrote: Just saw Dick's email and I guess I will push this rock a little further myself and start attending the FT calls if I can. Graham, Richard L. wrote: Dick, What are your objections here ? All the current proposal is doing is trying to define a set of consistent return codes, and is not changing anything about MPI. I am not sure there is sufficient information to act on in all cases with the current error handling in MPI, but may be wrong on this. However, there is nothing else that is being proposed at this stage. My main objection is that what I heard been discussed does not seem to add value. [rich] Yes, on it's own it adds little value, but not totally useless, but this is not an end of its own. Both you and Richard missed the context of the MPI Forum - not sure why you could not pick this up from the ether :-) Also, if no changes are made to current implementations, there is a chance that apps will hang, but this is no different than if users set errors return today. Which would need to do be done to see the effects of the changes. Now, it is fair to ask if this really adds something. If there is intent to recover from errors, which is what the FT working group is trying to figure out, then this has a lot of value in that it is the venue for an initial discussion of how to extend the error classes, which is really what Josh has been trying to do. I believe we need an implementation and some app experiments to find what we have missed. If this is being proposed just to bootstrap the discussion then fine I think, at least for the group, you've gotten some traction here ;-). :-) As for the choice of "MPI_ERR_CANNOT_CONTINUE" how is this any different than malloc returning null ? The problem as I see it is there might be very few times a library is completely out of commission and probably fewer times than that when the library actually know it is completely out of commission. Now maybe the point here is that this acts as a bridge between the current state of a library that supports MPI 2.2 to when a library supports all of the FT parts of MPI 3.0. That is, as I mentioned before, all libraries that have questions as to there FT support should automatically start returning MPI_ERR_CANNOT_CONTINUE after most/every error until they can fully support FT. [rich] Yes, this is the point. For the sort of environment I work in, the network stack returns sufficient information so that the library can know what is going on. The supporting run-times are also used to detect/notify when process fail, so the library can at least detect state. Responding does involve some changes to the library. I don't think it is unreasonable to respond with a "non-fatal" error code when resource requests can't be satisfied. Ok to be fair, there are probably libraries (OMPI) that do not necessarily handle the loss of a connection very well in certain cases and you could possibly mark a global to say the MPI library is borked in that case. However, if I was going to map out supporting MPI 3.0 FT I'd probably implement the whole thing taking into consideration these cases and returning the appropriate error instead of relying on the MPI_ERR_CANNOT_CONTINUE big hammer. [rich] Not sure that I would mark global state in this manner. But this really depends on how far the implementation wants to go just for this change. I believe the current proposal is a starting point for both the app and the library to start addressing the issues associated with failure. I expect that once you get into the guts of the library, and try to get beyond the equivalent of one big lock for threading, a pile of implementation issues will come up, and this is where the real value of this proposal is. It tells the user that the library is no longer functional, and leaves it to the app to decide how to respond. Depending on the implementation, there are "error" scenarios that both the app and the MPI library can survive. Failure of alloc_mem may be such a function. An app may also decide that sending data to a specific destination may also be ok - I can give a several real use cases that were brought to us as we were looking into this that would be just fine with this. Now the collective operations are another question. So, all this proposal is really doing is start to revive the FT discussion at the Forum level, as partial implementation is getting to a state that it can be evaluated. This is really why it is important to understand the specific shortcomings you see in what is being proposed - just the error propagation issues. Ok, so I see two issues off the bat mentioned: 1. Will an MPI library really know or ever want to throw MPI_ERR_CANNOT_CONTINUE. IMO, I think the answer should be no in that an MPI library should throw errors that are specific to whether a communicator or a connection to a rank is operable. Because I think there are no cases (except for bugs in the library) that a library is completely borked such that nothing can be done. I'll admit today that may not completely be true but in a 3.0 world shouldn't it? [rich] I think that experience will guide us here. I would tend to agree with you, if the library is aiming to recover from failures, but am reluctant to make a sweeping statement w/o more evidence. Libraries that do not want to support recovery are the candidates for returning such error codes. 2. The point Dick makes about the checking of a global value to determine if a library is borked could mess up cache performance is also something we should be concerned with. [rich] Agreed - I would not implement this in that manner. This is ok for some prototyping, but I would tend to implement things in a manner that when failure occurs, the library may do find out about the error deep down in the library, so that the common cause does not take the performance hit. I can actually see how to do this for pt-2-pt communications, but have not thought about collectives, for file ops. Rich --td Thanks, Rich On 9/22/10 6:55 PM, "Richard Treumann" <treumann(a)us.ibm.com> <mailto:[email protected]> wrote: We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. My argument is against adding it into the standard first and figuring out later what it means. I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. Note 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> <mailto:[email protected]> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> <mailto:[email protected]> Date: 09/22/2010 05:47 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: Process can no longer perform any MPI operations. If an MPI operation returns this error class, all subsequent calls to MPI functions will return this error class. Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. -d _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Terry Dontje 23 Sep '10

23 Sep '10

Just saw Dick's email and I guess I will push this rock a little further myself and start attending the FT calls if I can. Graham, Richard L. wrote: > Dick, > What are your objections here ? All the current proposal is doing is trying to define a set of consistent return codes, and is not changing anything about MPI. I am not sure there is sufficient information to act on in all cases with the current error handling in MPI, but may be wrong on this. However, there is nothing else that is being proposed at this stage. > My main objection is that what I heard been discussed does not seem to add value. > Also, if no changes are made to current implementations, there is a chance that apps will hang, but this is no different than if users set errors return today. Which would need to do be done to see the effects of the changes. > Now, it is fair to ask if this really adds something. If there is intent to recover from errors, which is what the FT working group is trying to figure out, then this has a lot of value in that it is the venue for an initial discussion of how to extend the error classes, which is really what Josh has been trying to do. I believe we need an implementation and some app experiments to find what we have missed. > If this is being proposed just to bootstrap the discussion then fine I think, at least for the group, you've gotten some traction here ;-). > As for the choice of "MPI_ERR_CANNOT_CONTINUE" how is this any different than malloc returning null ? The problem as I see it is there might be very few times a library is completely out of commission and probably fewer times than that when the library actually know it is completely out of commission. Now maybe the point here is that this acts as a bridge between the current state of a library that supports MPI 2.2 to when a library supports all of the FT parts of MPI 3.0. That is, as I mentioned before, all libraries that have questions as to there FT support should automatically start returning MPI_ERR_CANNOT_CONTINUE after most/every error until they can fully support FT. Ok to be fair, there are probably libraries (OMPI) that do not necessarily handle the loss of a connection very well in certain cases and you could possibly mark a global to say the MPI library is borked in that case. However, if I was going to map out supporting MPI 3.0 FT I'd probably implement the whole thing taking into consideration these cases and returning the appropriate error instead of relying on the MPI_ERR_CANNOT_CONTINUE big hammer. > It tells the user that the library is no longer functional, and leaves it to the app to decide how to respond. Depending on the implementation, there are "error" scenarios that both the app and the MPI library can survive. Failure of alloc_mem may be such a function. An app may also decide that sending data to a specific destination may also be ok - I can give a several real use cases that were brought to us as we were looking into this that would be just fine with this. Now the collective operations are another question. > So, all this proposal is really doing is start to revive the FT discussion at the Forum level, as partial implementation is getting to a state that it can be evaluated. This is really why it is important to understand the specific shortcomings you see in what is being proposed - just the error propagation issues. > > Ok, so I see two issues off the bat mentioned: 1. Will an MPI library really know or ever want to throw MPI_ERR_CANNOT_CONTINUE. IMO, I think the answer should be no in that an MPI library should throw errors that are specific to whether a communicator or a connection to a rank is operable. Because I think there are no cases (except for bugs in the library) that a library is completely borked such that nothing can be done. I'll admit today that may not completely be true but in a 3.0 world shouldn't it? 2. The point Dick makes about the checking of a global value to determine if a library is borked could mess up cache performance is also something we should be concerned with. --td > Thanks, > Rich > > On 9/22/10 6:55 PM, "Richard Treumann" <treumann(a)us.ibm.com> wrote: > > > We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. > > My argument is against adding it into the standard first and figuring out later what it means. > > I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. > > If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. > > Note > 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. > 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Darius Buntinas <buntinas(a)mcs.anl.gov> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/22/2010 05:47 PM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > ________________________________ > > > > > On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: > > >> You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? >> > > Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) > > >> Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? >> > > I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: > > Process can no longer perform any MPI operations. If an MPI operation > returns this error class, all subsequent calls to MPI functions will > return this error class. > > >> Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. >> > > Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. > > OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. > > So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. > > -d > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Oracle Terry D. Dontje | Principal Software Engineer Developer Tools Engineering | +1.781.442.2631 Oracle * - Performance Technologies* 95 Network Drive, Burlington, MA 01803 Email terry.dontje(a)oracle.com <mailto:[email protected]>

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Graham, Richard L. 23 Sep '10

23 Sep '10

Please see comment in line: On 9/22/10 11:41 PM, "Richard Treumann" <treumann(a)us.ibm.com> wrote: Rich I am not on the FT working group. I do not know the ins and out of your ultimate goals or how you hope they can be accomplished. I do not intend to become immersed in the complete FT specification. I cannot afford the time. Those who are immersed seem to feel they have a "minor" tweak to add to the standard before they have the rest of FT worked out. It does not sound "minor" to me but maybe I have misunderstood some part of it. I think the logical thing for me to do is wait for the ticket. To be voted, the ticket must completely specify the functionality and include the precise text to explain what it means. If, by the time you have a ticket, my concerns are resolved by what it says, I will not object. If I am still concerned by what the ticket proposes, I must lobby against it and at that point I can stick to the merits or risks contained in the ticket language only. At this point, I am trying to discuss the shape of a cloud of smoke and the discussion has become unproductive. I have tried several ways of making my concern understandable so I do not know what to make of the question "What are your objections here?" at this stage. [rich] What I am trying to figure out what is not palatable here. Is it that the change is not a small as it seems, is it that on it's own, this really does not add much, is that it is not sufficiently worked out at this stage, or is it something else ? As a working group we would like to make sure to address concerns. Thanks for all the comments - these are appreciated, Rich I understand that something like MPI_ERR_CANNOT_CONTINUE could possibly make sense in the context of a complete FT chapter that devotes tens of pages to saying exactly when it is appropriate to return and how it is to be understood by people who wish to write portable MPI applications. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: "Graham, Richard L." <rlgraham(a)ornl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 08:52 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ Dick, What are your objections here ? All the current proposal is doing is trying to define a set of consistent return codes, and is not changing anything about MPI. I am not sure there is sufficient information to act on in all cases with the current error handling in MPI, but may be wrong on this. However, there is nothing else that is being proposed at this stage. Also, if no changes are made to current implementations, there is a chance that apps will hang, but this is no different than if users set errors return today. Which would need to do be done to see the effects of the changes. Now, it is fair to ask if this really adds something. If there is intent to recover from errors, which is what the FT working group is trying to figure out, then this has a lot of value in that it is the venue for an initial discussion of how to extend the error classes, which is really what Josh has been trying to do. I believe we need an implementation and some app experiments to find what we have missed. As for the choice of "MPI_ERR_CANNOT_CONTINUE" how is this any different than malloc returning null ? It tells the user that the library is no longer functional, and leaves it to the app to decide how to respond. Depending on the implementation, there are "error" scenarios that both the app and the MPI library can survive. Failure of alloc_mem may be such a function. An app may also decide that sending data to a specific destination may also be ok - I can give a several real use cases that were brought to us as we were looking into this that would be just fine with this. Now the collective operations are another question. So, all this proposal is really doing is start to revive the FT discussion at the Forum level, as partial implementation is getting to a state that it can be evaluated. This is really why it is important to understand the specific shortcomings you see in what is being proposed - just the error propagation issues. Thanks, Rich On 9/22/10 6:55 PM, "Richard Treumann" <treumann(a)us.ibm.com> wrote: We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. My argument is against adding it into the standard first and figuring out later what it means. I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. Note 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 05:47 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: > > You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) > Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: Process can no longer perform any MPI operations. If an MPI operation returns this error class, all subsequent calls to MPI functions will return this error class. > Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. -d _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft>

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 22 Sep '10

22 Sep '10

Rich I am not on the FT working group. I do not know the ins and out of your ultimate goals or how you hope they can be accomplished. I do not intend to become immersed in the complete FT specification. I cannot afford the time. Those who are immersed seem to feel they have a "minor" tweak to add to the standard before they have the rest of FT worked out. It does not sound "minor" to me but maybe I have misunderstood some part of it. I think the logical thing for me to do is wait for the ticket. To be voted, the ticket must completely specify the functionality and include the precise text to explain what it means. If, by the time you have a ticket, my concerns are resolved by what it says, I will not object. If I am still concerned by what the ticket proposes, I must lobby against it and at that point I can stick to the merits or risks contained in the ticket language only. At this point, I am trying to discuss the shape of a cloud of smoke and the discussion has become unproductive. I have tried several ways of making my concern understandable so I do not know what to make of the question "What are your objections here?" at this stage. I understand that something like MPI_ERR_CANNOT_CONTINUE could possibly make sense in the context of a complete FT chapter that devotes tens of pages to saying exactly when it is appropriate to return and how it is to be understood by people who wish to write portable MPI applications. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: "Graham, Richard L." <rlgraham(a)ornl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 08:52 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org Dick, What are your objections here ? All the current proposal is doing is trying to define a set of consistent return codes, and is not changing anything about MPI. I am not sure there is sufficient information to act on in all cases with the current error handling in MPI, but may be wrong on this. However, there is nothing else that is being proposed at this stage. Also, if no changes are made to current implementations, there is a chance that apps will hang, but this is no different than if users set errors return today. Which would need to do be done to see the effects of the changes. Now, it is fair to ask if this really adds something. If there is intent to recover from errors, which is what the FT working group is trying to figure out, then this has a lot of value in that it is the venue for an initial discussion of how to extend the error classes, which is really what Josh has been trying to do. I believe we need an implementation and some app experiments to find what we have missed. As for the choice of "MPI_ERR_CANNOT_CONTINUE" how is this any different than malloc returning null ? It tells the user that the library is no longer functional, and leaves it to the app to decide how to respond. Depending on the implementation, there are "error" scenarios that both the app and the MPI library can survive. Failure of alloc_mem may be such a function. An app may also decide that sending data to a specific destination may also be ok - I can give a several real use cases that were brought to us as we were looking into this that would be just fine with this. Now the collective operations are another question. So, all this proposal is really doing is start to revive the FT discussion at the Forum level, as partial implementation is getting to a state that it can be evaluated. This is really why it is important to understand the specific shortcomings you see in what is being proposed - just the error propagation issues. Thanks, Rich On 9/22/10 6:55 PM, "Richard Treumann" <treumann(a)us.ibm.com> wrote: We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. My argument is against adding it into the standard first and figuring out later what it means. I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. Note 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 05:47 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: > > You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) > Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: Process can no longer perform any MPI operations. If an MPI operation returns this error class, all subsequent calls to MPI functions will return this error class. > Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. -d _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft < http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Graham, Richard L. 22 Sep '10

22 Sep '10

Dick, What are your objections here ? All the current proposal is doing is trying to define a set of consistent return codes, and is not changing anything about MPI. I am not sure there is sufficient information to act on in all cases with the current error handling in MPI, but may be wrong on this. However, there is nothing else that is being proposed at this stage. Also, if no changes are made to current implementations, there is a chance that apps will hang, but this is no different than if users set errors return today. Which would need to do be done to see the effects of the changes. Now, it is fair to ask if this really adds something. If there is intent to recover from errors, which is what the FT working group is trying to figure out, then this has a lot of value in that it is the venue for an initial discussion of how to extend the error classes, which is really what Josh has been trying to do. I believe we need an implementation and some app experiments to find what we have missed. As for the choice of "MPI_ERR_CANNOT_CONTINUE" how is this any different than malloc returning null ? It tells the user that the library is no longer functional, and leaves it to the app to decide how to respond. Depending on the implementation, there are "error" scenarios that both the app and the MPI library can survive. Failure of alloc_mem may be such a function. An app may also decide that sending data to a specific destination may also be ok - I can give a several real use cases that were brought to us as we were looking into this that would be just fine with this. Now the collective operations are another question. So, all this proposal is really doing is start to revive the FT discussion at the Forum level, as partial implementation is getting to a state that it can be evaluated. This is really why it is important to understand the specific shortcomings you see in what is being proposed - just the error propagation issues. Thanks, Rich On 9/22/10 6:55 PM, "Richard Treumann" <treumann(a)us.ibm.com> wrote: We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. My argument is against adding it into the standard first and figuring out later what it means. I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. Note 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 05:47 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: > > You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) > Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: Process can no longer perform any MPI operations. If an MPI operation returns this error class, all subsequent calls to MPI functions will return this error class. > Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. -d _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft>

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 22 Sep '10

22 Sep '10

We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. My argument is against adding it into the standard first and figuring out later what it means. I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. Note 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 05:47 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: > > You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) > Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: Process can no longer perform any MPI operations. If an MPI operation returns this error class, all subsequent calls to MPI functions will return this error class. > Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. -d _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Darius Buntinas 22 Sep '10

22 Sep '10

On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: > > You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) > Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: Process can no longer perform any MPI operations. If an MPI operation returns this error class, all subsequent calls to MPI functions will return this error class. > Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. -d

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 22 Sep '10

22 Sep '10

An incident that trashes MPI internal data structures will seldom be recognizable as a trigger for a non_SUCCESS rc on a particular call. If you do an MPI_Recv and the buffer pointer happens to drop the data all over MPI internal state, there is about zero chance the MPI_Recv call will be able to detect that. It will just return MPI_SUCCESS and depending on what you trashed, things may run fine or may break in unpredictable ways. I have absolutely no problem with a community agreement to try some prototyping of ideas that will be proposed for the standard if they prove out. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: "Bronevetsky, Greg" <bronevetsky1(a)llnl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 03:37 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org One candidate for CANNOT_CONTINUE would be a data corruption in MPI memory or some data structure inconsistency due to a bug. This could have zero effect or could corrupt application results or system state. It would be exceedingly difficult for MPI to do anything meaningful here and continued operation is potentially very dangerous. As such, I would consider this to be a bad enough error to return CANNOT_CONTINUE. I think the point of this proposal is not that CANNOT_CONTINUE is going to be a common error but to lay the groundwork for a more useful error reporting scheme. Today we’re quite sure that CANNOT_CONTINUE will be the worst thing that an MPI implementation will want to return but we’re not really sure about what the other errors will look like. For example, RANK_DEAD and LINK_DEAD sound like plausible error messages but we won’t know until individual implementations have had a chance to experiment them. This proposal allows such experimentation to happen within the same basic error reporting framework. Having said that, I’m not completely convinced that we need to include this in the spec yet or whether this can be more like a community agreement until we understand the problem better. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com From: mpi3-ft-bounces(a)lists.mpi-forum.org [ mailto:[email protected]] On Behalf Of Richard Treumann Sent: Wednesday, September 22, 2010 11:17 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Darius I can imagine a few errors that I know will be harmless to MPI state. I can make sure nobody can do any harm by passing an invalid communicator to MPI_COMM_SIZE. I cannot think of a detectable error that would return and leave that thread of that process so totally broken that nothing in MPI will work from then on. In a collective, there may be processes in which the thread that called the CC never returns and that tread of the process is no longer usable because it is hung. Other threads using other communicators in the process with a hung thread may work perfectly. Except for the very few cases where I know there was no damage (like a bad comm on MPI_COMM_SIZE) the situation, 99.99% of the time, will be that everything still works but sometimes the outcome is a surprise to the user. Say you do: 1 MPI_Barrier (on world) 2 MPI_Barrier (on world): 3 other stuff 4 MPI_Barrier (on world) 5 if (my rank is even) 6 sendrecv(with odd neighbor) 7 else 8 sendrecv(with even neighbor) but get back an error at all even numbered ranks from the line 1 barrier call. The line 2 MPI_Barrier may still "work" but the line 2 barrier at even numbered ranks will match the line 1 barrier at odd ranks. Even ranks will begin "other stuff" and odd ranks will sit in the line 2 barrier until even ranks finish "other stuff" and reach the line 4 barrier. The odd ranks now get through their line 2 barrier and begin other stuff. If "other stuff" involves communication among the even ranks and communication among the odd ranks. that will work too. The even ranks will all send/recv among themselves later the odd ranks will all send/recv among themselves. The even ranks will reach line 6 and hang there because the odd tasks are still stuck at line 4. In this entire example, libmpi has continued working "correctly" but the behavior you get from correct behavior is not what you planned. The situation of MPI state being totally trashed by an error that returns a return code barely exists. The case where it is subtly discombobulated is the norm. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 12:24 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org OK, I (think I) see what you guys are saying, so maybe we should look at it this way. The CANNOT_CONTINUE proposal should not define the operation of the MPI implementation after errors other than CANNOT_CONTINUE. Instead, it defines that after the implementation gives a CANNOT_CONTINUE error, the app knows that the implementation is fatally wedged, and that the user should definately not expect correct operation after this. I.e., we're not labeling other errors as "recoverable," we're just marking CANNOT_CONTINUE as "unrecoverable." Note that an implementation can still be standard compliant even if it never returns a CANNOT_CONTINUE error even when it is fatally wedged (because operation after any other error is still undefined). This just defines a way for the implementation to let the user know that it has given up. So that if the implementation provides best-effort functionality after an error, and the user has "read the disclaimer" and is comfortable with proceeding, this is a way to differentiate between an error as a result of a failure that hosed everything, and one that may allow things to continue. We still would like to define what happens to a bcast after a process in the communicator fails. But we leave that for future proposals. Does this make sense? -d On Sep 22, 2010, at 8:43 AM, Terry Dontje wrote: > Richard Treumann wrote: >> >> This proposal is not a minor change. >> >> Please do not make this hole in the standard and assume you can later add language to standardize everything that comes through the hole. >> >> If the standard is to introduce the notion of a recoverable error it must be as part of a full description of what "recovery" means. >> >> I think is is dangerous and ultimately useless to have implementors mark a failure as "recoverable" when the post error state of the distributed MPI has gone from "fully standards compliant" to "mostly standards compliant, read my user doc read my legal disclaimer, cross your fingers". >> >> See comment below for why I do not think the new hole is needed to allow people to do implementation specific recoverability. >> >> There is not even anything to prevent on implementation from deciding to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and documenting 5 or 5000 enumerated values for "answer" ranging from NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. >> >> IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot imagine how we would promise exactly what will work and what will not but in practice most things will still work as expected. >> > I think I agree with Dick on the above. Another way of putting the disagreement is that Josh's proposal is too general in that not all errorcodes can be completely marked as MPI state is broken or not. When Sun implemented fault tolerant client/server we came up with a new error class that when returned gave the user the understanding that a condition occurred on a communicator that has rendered the communicator useless and one should clean it up before continuing on. The point is there was a concrete understanding of the error and what could be done to recover. As opposed to a general class that say's everything is borked or not which essential doesn't give you much because you'll end up eventually having to define a more specific class of error IMO. > > --td >> >> >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: >> >> > [image removed] >> > >> > Re: [Mpi3-ft] Defining the state of MPI after an error >> > >> > Bronis R. de Supinski >> > >> > to: >> > >> > MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> > >> > 09/21/2010 04:59 PM >> > >> > Sent by: >> > >> > mpi3-ft-bounces(a)lists.mpi-forum.org >> > >> > Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance >> > and Dynamic Process Control working Group" >> > >> > >> > Dick: >> > >> > Re: >> > > The current MPI standard does not say the MPI implementation is totally >> > > broken once there is an error. Saying MPI state is undefined after an >> > > error simply says that the detailed semantic of the MPI standard can no >> > > longer be promised. In other words, after an error you leave behind the >> > > security of a portable standard semantic. You are operating at your own >> > > risk. You do not need to read more than that into it. >> > >> > Perhaps my problem with this position is that I come from the >> > background of language definitions for compilers. When you >> > read "undefined" in the OpenMP specification then you are >> > being told that things are broken and the implementation does >> > need to do anything or even tell you what they actually do (and >> > I believe the same is true for the C and C++ standards). An >> > alternative is "implementation defined", which requires the >> > implementer to document what they actually do. Without that, >> > you cannot even rely on actions with a specific implementation >> > (unless you believe "My tests so far have not failed so I am OK"). >> >> >> When a standard says behavior is "undefined" in some situation, it cannot mean behavior is "broken". It cannot mean the implementor is prohibited from making it still work. It cannot mean the implementor is prohibited from making certain things work and documenting them. Any statement like this in a standard would be definition of behavior and the behavior would no longer be "undefined". >> >> The only thing a standard can logically mean by "undefined" is that the STANDARD no longer mandates the definition. >> >> Bronis says: >> >> > >> > I strongly feel "undefined" should be reserved for situations that >> > mean "your program is irrevocably broken and the implementer does >> > not need to worry about what happens to it after encountering them." >> >> I would say this as: >> >> I strongly feel "undefined" should be reserved for situations that mean "The standard no longer guarantees your program is not irrevocably broken. The implementer is not required by the standard to worry about what happens to it after encountering them. An Implementation is free to provide any "better" behavior that may be of value but users cannot assume another implementation provides similar behavior so cannot assume standards defined portability." >> >> I do not see how the use if the word "undefined" in a standard can be interpreted as a prohibition of any behavior an implementation might offer. >> >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> > > > -- > <Mail Attachment.gif> > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle - Performance Technologies > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Graham, Richard L. 22 Sep '10

22 Sep '10

In addition, if there will be returned something other than CANNOT_CONTINUE there needs to be a mechanism to return information beyond just a single integer. In many situations this is sufficient, but not in all. As a simple example, in earlier discussions there was consideration to what would happen if a wild card source receive was posted, and a member of the communicator failed. We had agreed that would put the communicator into an error state, with the library not being able to guess the app's intent, which would require an error to be returned. At this stage, the app may need to know which rank failed, as it can't infer this from the error code who failed, and would need a mechanism to get this information. Rich On 9/22/10 3:36 PM, "Bronevetsky, Greg" <bronevetsky1(a)llnl.gov> wrote: One candidate for CANNOT_CONTINUE would be a data corruption in MPI memory or some data structure inconsistency due to a bug. This could have zero effect or could corrupt application results or system state. It would be exceedingly difficult for MPI to do anything meaningful here and continued operation is potentially very dangerous. As such, I would consider this to be a bad enough error to return CANNOT_CONTINUE. I think the point of this proposal is not that CANNOT_CONTINUE is going to be a common error but to lay the groundwork for a more useful error reporting scheme. Today we're quite sure that CANNOT_CONTINUE will be the worst thing that an MPI implementation will want to return but we're not really sure about what the other errors will look like. For example, RANK_DEAD and LINK_DEAD sound like plausible error messages but we won't know until individual implementations have had a chance to experiment them. This proposal allows such experimentation to happen within the same basic error reporting framework. Having said that, I'm not completely convinced that we need to include this in the spec yet or whether this can be more like a community agreement until we understand the problem better. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Treumann Sent: Wednesday, September 22, 2010 11:17 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Darius I can imagine a few errors that I know will be harmless to MPI state. I can make sure nobody can do any harm by passing an invalid communicator to MPI_COMM_SIZE. I cannot think of a detectable error that would return and leave that thread of that process so totally broken that nothing in MPI will work from then on. In a collective, there may be processes in which the thread that called the CC never returns and that tread of the process is no longer usable because it is hung. Other threads using other communicators in the process with a hung thread may work perfectly. Except for the very few cases where I know there was no damage (like a bad comm on MPI_COMM_SIZE) the situation, 99.99% of the time, will be that everything still works but sometimes the outcome is a surprise to the user. Say you do: 1 MPI_Barrier (on world) 2 MPI_Barrier (on world): 3 other stuff 4 MPI_Barrier (on world) 5 if (my rank is even) 6 sendrecv(with odd neighbor) 7 else 8 sendrecv(with even neighbor) but get back an error at all even numbered ranks from the line 1 barrier call. The line 2 MPI_Barrier may still "work" but the line 2 barrier at even numbered ranks will match the line 1 barrier at odd ranks. Even ranks will begin "other stuff" and odd ranks will sit in the line 2 barrier until even ranks finish "other stuff" and reach the line 4 barrier. The odd ranks now get through their line 2 barrier and begin other stuff. If "other stuff" involves communication among the even ranks and communication among the odd ranks. that will work too. The even ranks will all send/recv among themselves later the odd ranks will all send/recv among themselves. The even ranks will reach line 6 and hang there because the odd tasks are still stuck at line 4. In this entire example, libmpi has continued working "correctly" but the behavior you get from correct behavior is not what you planned. The situation of MPI state being totally trashed by an error that returns a return code barely exists. The case where it is subtly discombobulated is the norm. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 12:24 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ OK, I (think I) see what you guys are saying, so maybe we should look at it this way. The CANNOT_CONTINUE proposal should not define the operation of the MPI implementation after errors other than CANNOT_CONTINUE. Instead, it defines that after the implementation gives a CANNOT_CONTINUE error, the app knows that the implementation is fatally wedged, and that the user should definately not expect correct operation after this. I.e., we're not labeling other errors as "recoverable," we're just marking CANNOT_CONTINUE as "unrecoverable." Note that an implementation can still be standard compliant even if it never returns a CANNOT_CONTINUE error even when it is fatally wedged (because operation after any other error is still undefined). This just defines a way for the implementation to let the user know that it has given up. So that if the implementation provides best-effort functionality after an error, and the user has "read the disclaimer" and is comfortable with proceeding, this is a way to differentiate between an error as a result of a failure that hosed everything, and one that may allow things to continue. We still would like to define what happens to a bcast after a process in the communicator fails. But we leave that for future proposals. Does this make sense? -d On Sep 22, 2010, at 8:43 AM, Terry Dontje wrote: > Richard Treumann wrote: >> >> This proposal is not a minor change. >> >> Please do not make this hole in the standard and assume you can later add language to standardize everything that comes through the hole. >> >> If the standard is to introduce the notion of a recoverable error it must be as part of a full description of what "recovery" means. >> >> I think is is dangerous and ultimately useless to have implementors mark a failure as "recoverable" when the post error state of the distributed MPI has gone from "fully standards compliant" to "mostly standards compliant, read my user doc read my legal disclaimer, cross your fingers". >> >> See comment below for why I do not think the new hole is needed to allow people to do implementation specific recoverability. >> >> There is not even anything to prevent on implementation from deciding to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and documenting 5 or 5000 enumerated values for "answer" ranging from NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. >> >> IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot imagine how we would promise exactly what will work and what will not but in practice most things will still work as expected. >> > I think I agree with Dick on the above. Another way of putting the disagreement is that Josh's proposal is too general in that not all errorcodes can be completely marked as MPI state is broken or not. When Sun implemented fault tolerant client/server we came up with a new error class that when returned gave the user the understanding that a condition occurred on a communicator that has rendered the communicator useless and one should clean it up before continuing on. The point is there was a concrete understanding of the error and what could be done to recover. As opposed to a general class that say's everything is borked or not which essential doesn't give you much because you'll end up eventually having to define a more specific class of error IMO. > > --td >> >> >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: >> >> > [image removed] >> > >> > Re: [Mpi3-ft] Defining the state of MPI after an error >> > >> > Bronis R. de Supinski >> > >> > to: >> > >> > MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> > >> > 09/21/2010 04:59 PM >> > >> > Sent by: >> > >> > mpi3-ft-bounces(a)lists.mpi-forum.org >> > >> > Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance >> > and Dynamic Process Control working Group" >> > >> > >> > Dick: >> > >> > Re: >> > > The current MPI standard does not say the MPI implementation is totally >> > > broken once there is an error. Saying MPI state is undefined after an >> > > error simply says that the detailed semantic of the MPI standard can no >> > > longer be promised. In other words, after an error you leave behind the >> > > security of a portable standard semantic. You are operating at your own >> > > risk. You do not need to read more than that into it. >> > >> > Perhaps my problem with this position is that I come from the >> > background of language definitions for compilers. When you >> > read "undefined" in the OpenMP specification then you are >> > being told that things are broken and the implementation does >> > need to do anything or even tell you what they actually do (and >> > I believe the same is true for the C and C++ standards). An >> > alternative is "implementation defined", which requires the >> > implementer to document what they actually do. Without that, >> > you cannot even rely on actions with a specific implementation >> > (unless you believe "My tests so far have not failed so I am OK"). >> >> >> When a standard says behavior is "undefined" in some situation, it cannot mean behavior is "broken". It cannot mean the implementor is prohibited from making it still work. It cannot mean the implementor is prohibited from making certain things work and documenting them. Any statement like this in a standard would be definition of behavior and the behavior would no longer be "undefined". >> >> The only thing a standard can logically mean by "undefined" is that the STANDARD no longer mandates the definition. >> >> Bronis says: >> >> > >> > I strongly feel "undefined" should be reserved for situations that >> > mean "your program is irrevocably broken and the implementer does >> > not need to worry about what happens to it after encountering them." >> >> I would say this as: >> >> I strongly feel "undefined" should be reserved for situations that mean "The standard no longer guarantees your program is not irrevocably broken. The implementer is not required by the standard to worry about what happens to it after encountering them. An Implementation is free to provide any "better" behavior that may be of value but users cannot assume another implementation provides similar behavior so cannot assume standards defined portability." >> >> I do not see how the use if the word "undefined" in a standard can be interpreted as a prohibition of any behavior an implementation might offer. >> >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >> >> >> > > > -- > <Mail Attachment.gif> > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle - Performance Technologies > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft>

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Bronevetsky, Greg 22 Sep '10

22 Sep '10

One candidate for CANNOT_CONTINUE would be a data corruption in MPI memory or some data structure inconsistency due to a bug. This could have zero effect or could corrupt application results or system state. It would be exceedingly difficult for MPI to do anything meaningful here and continued operation is potentially very dangerous. As such, I would consider this to be a bad enough error to return CANNOT_CONTINUE. I think the point of this proposal is not that CANNOT_CONTINUE is going to be a common error but to lay the groundwork for a more useful error reporting scheme. Today we're quite sure that CANNOT_CONTINUE will be the worst thing that an MPI implementation will want to return but we're not really sure about what the other errors will look like. For example, RANK_DEAD and LINK_DEAD sound like plausible error messages but we won't know until individual implementations have had a chance to experiment them. This proposal allows such experimentation to happen within the same basic error reporting framework. Having said that, I'm not completely convinced that we need to include this in the spec yet or whether this can be more like a community agreement until we understand the problem better. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Treumann Sent: Wednesday, September 22, 2010 11:17 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Darius I can imagine a few errors that I know will be harmless to MPI state. I can make sure nobody can do any harm by passing an invalid communicator to MPI_COMM_SIZE. I cannot think of a detectable error that would return and leave that thread of that process so totally broken that nothing in MPI will work from then on. In a collective, there may be processes in which the thread that called the CC never returns and that tread of the process is no longer usable because it is hung. Other threads using other communicators in the process with a hung thread may work perfectly. Except for the very few cases where I know there was no damage (like a bad comm on MPI_COMM_SIZE) the situation, 99.99% of the time, will be that everything still works but sometimes the outcome is a surprise to the user. Say you do: 1 MPI_Barrier (on world) 2 MPI_Barrier (on world): 3 other stuff 4 MPI_Barrier (on world) 5 if (my rank is even) 6 sendrecv(with odd neighbor) 7 else 8 sendrecv(with even neighbor) but get back an error at all even numbered ranks from the line 1 barrier call. The line 2 MPI_Barrier may still "work" but the line 2 barrier at even numbered ranks will match the line 1 barrier at odd ranks. Even ranks will begin "other stuff" and odd ranks will sit in the line 2 barrier until even ranks finish "other stuff" and reach the line 4 barrier. The odd ranks now get through their line 2 barrier and begin other stuff. If "other stuff" involves communication among the even ranks and communication among the odd ranks. that will work too. The even ranks will all send/recv among themselves later the odd ranks will all send/recv among themselves. The even ranks will reach line 6 and hang there because the odd tasks are still stuck at line 4. In this entire example, libmpi has continued working "correctly" but the behavior you get from correct behavior is not what you planned. The situation of MPI state being totally trashed by an error that returns a return code barely exists. The case where it is subtly discombobulated is the norm. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/22/2010 12:24 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ OK, I (think I) see what you guys are saying, so maybe we should look at it this way. The CANNOT_CONTINUE proposal should not define the operation of the MPI implementation after errors other than CANNOT_CONTINUE. Instead, it defines that after the implementation gives a CANNOT_CONTINUE error, the app knows that the implementation is fatally wedged, and that the user should definately not expect correct operation after this. I.e., we're not labeling other errors as "recoverable," we're just marking CANNOT_CONTINUE as "unrecoverable." Note that an implementation can still be standard compliant even if it never returns a CANNOT_CONTINUE error even when it is fatally wedged (because operation after any other error is still undefined). This just defines a way for the implementation to let the user know that it has given up. So that if the implementation provides best-effort functionality after an error, and the user has "read the disclaimer" and is comfortable with proceeding, this is a way to differentiate between an error as a result of a failure that hosed everything, and one that may allow things to continue. We still would like to define what happens to a bcast after a process in the communicator fails. But we leave that for future proposals. Does this make sense? -d On Sep 22, 2010, at 8:43 AM, Terry Dontje wrote: > Richard Treumann wrote: >> >> This proposal is not a minor change. >> >> Please do not make this hole in the standard and assume you can later add language to standardize everything that comes through the hole. >> >> If the standard is to introduce the notion of a recoverable error it must be as part of a full description of what "recovery" means. >> >> I think is is dangerous and ultimately useless to have implementors mark a failure as "recoverable" when the post error state of the distributed MPI has gone from "fully standards compliant" to "mostly standards compliant, read my user doc read my legal disclaimer, cross your fingers". >> >> See comment below for why I do not think the new hole is needed to allow people to do implementation specific recoverability. >> >> There is not even anything to prevent on implementation from deciding to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and documenting 5 or 5000 enumerated values for "answer" ranging from NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. >> >> IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot imagine how we would promise exactly what will work and what will not but in practice most things will still work as expected. >> > I think I agree with Dick on the above. Another way of putting the disagreement is that Josh's proposal is too general in that not all errorcodes can be completely marked as MPI state is broken or not. When Sun implemented fault tolerant client/server we came up with a new error class that when returned gave the user the understanding that a condition occurred on a communicator that has rendered the communicator useless and one should clean it up before continuing on. The point is there was a concrete understanding of the error and what could be done to recover. As opposed to a general class that say's everything is borked or not which essential doesn't give you much because you'll end up eventually having to define a more specific class of error IMO. > > --td >> >> >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: >> >> > [image removed] >> > >> > Re: [Mpi3-ft] Defining the state of MPI after an error >> > >> > Bronis R. de Supinski >> > >> > to: >> > >> > MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> > >> > 09/21/2010 04:59 PM >> > >> > Sent by: >> > >> > mpi3-ft-bounces(a)lists.mpi-forum.org >> > >> > Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance >> > and Dynamic Process Control working Group" >> > >> > >> > Dick: >> > >> > Re: >> > > The current MPI standard does not say the MPI implementation is totally >> > > broken once there is an error. Saying MPI state is undefined after an >> > > error simply says that the detailed semantic of the MPI standard can no >> > > longer be promised. In other words, after an error you leave behind the >> > > security of a portable standard semantic. You are operating at your own >> > > risk. You do not need to read more than that into it. >> > >> > Perhaps my problem with this position is that I come from the >> > background of language definitions for compilers. When you >> > read "undefined" in the OpenMP specification then you are >> > being told that things are broken and the implementation does >> > need to do anything or even tell you what they actually do (and >> > I believe the same is true for the C and C++ standards). An >> > alternative is "implementation defined", which requires the >> > implementer to document what they actually do. Without that, >> > you cannot even rely on actions with a specific implementation >> > (unless you believe "My tests so far have not failed so I am OK"). >> >> >> When a standard says behavior is "undefined" in some situation, it cannot mean behavior is "broken". It cannot mean the implementor is prohibited from making it still work. It cannot mean the implementor is prohibited from making certain things work and documenting them. Any statement like this in a standard would be definition of behavior and the behavior would no longer be "undefined". >> >> The only thing a standard can logically mean by "undefined" is that the STANDARD no longer mandates the definition. >> >> Bronis says: >> >> > >> > I strongly feel "undefined" should be reserved for situations that >> > mean "your program is irrevocably broken and the implementer does >> > not need to worry about what happens to it after encountering them." >> >> I would say this as: >> >> I strongly feel "undefined" should be reserved for situations that mean "The standard no longer guarantees your program is not irrevocably broken. The implementer is not required by the standard to worry about what happens to it after encountering them. An Implementation is free to provide any "better" behavior that may be of value but users cannot assume another implementation provides similar behavior so cannot assume standards defined portability." >> >> I do not see how the use if the word "undefined" in a standard can be interpreted as a prohibition of any behavior an implementation might offer. >> >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> > > > -- > <Mail Attachment.gif> > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle - Performance Technologies > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0