September 2010 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 30 Sep '10

30 Sep '10

Josh & the FT Team Thanks for this decision to suspend this ticket for now. I am not in any position to judge whether the idea can be part of an integrated approach to Fault Tolerance. If it, or something similar, comes up in the context of an FT chapter and is well defined in that context, I would not expect to have any objection. I expect the discussion of any FT chapter brought to the Forum as a whole to be difficult but I hope we are generally able so see it as a whole that can be accepted as ready for a standards document. If not that, I hope it can be recognized as broadly promising but not mature enough to standardize. and moved to the Journal of R&D as a proposal that people can implement and refine before a later decision to standardize. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Joshua Hursey <jjhursey(a)open-mpi.org> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/29/2010 01:30 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org It was the feeling of the group during the teleconf that this proposal should be suspended for the time being. I drafted a 'proposal resolution' rationale distilled from all of the discussion so far and included it with the proposed text on the wiki. If you feel that something is missing or misrepresented in the resolution text, please let me know. This resolution is this text that we will point the supporters of this idea to when they ask about its status. The text can be found at the following link: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue#Pr… -- Josh On Sep 23, 2010, at 3:54 PM, Joshua Hursey wrote: > > On Sep 23, 2010, at 10:40 AM, Richard Treumann wrote: > >> >> A few quick observations: >> >> 0) >> The constant is MPI_ERROR_ARE_FATAL, not MPI_ERRORS_ABORT > > Fixed. Thanks > >> >> 1) >> The MPI standard only mandates one return code, MPI_SUCCESS. All other return codes are implementation specific and non-portable. For portability, MPI documents error classes and a query function that is passed an implementation defined return code and returns the class. >> >> Assume I allow tags between 0 and 2**15. As an MPI implementor, I am free to use return code 215 for a negative tag and 399 for one that is above 2**15. The error message I print for 215 and the error message I print for 399 can be different. If the user calls MPI_ERROR_CLASS() with either 215 or 399 I give back the class MPI_ARR_TAG. The user who checks the RC of a call to see if it is == MPI_ERR_TAG has written non-portable code. >> >> If I decide return codes 251 and 399 must be in class MPI_CANNOT_CONTINUE they can no longer be in class MPI_ERR_TAG. > > I understand, that is why MPI_ERR_CANNOT_CONTINUE is an error class. We are not defining a new error code, but a new error class. > >> >> >> 2) >> The MPI standard avoids mandating specific error checks. It identifies a lot of errors and in many cases, says what error class that error is in. It does not say an implementation MUST detect the error. I would not violate the standard by skipping the check of whether MPI is initialized. My customers may demand it but the standard does not. You are introducing a mandate for one specific sort of error. > > Yes that is covered in the opening paragraph of 8.3. We are not mandating that an implementation must detect errors, but that if it does detect an error it must return an error code in an error class. So no change in this regard from the existing standard. This proposal adds that if the MPI implementation decides that it cannot continue after it detects the original error then it must return an error code in the MPI_ERR_CANNOT_CONTINUE error class on subsequent MPI calls. So I guess you could say that it must detect that it is itself in an self-defined erroneous state in order to return MPI_ERR_CANNOT_CONTINUE. > > On a related note, the standard language does not specify how an implementation goes about implementing the MPI_ERR_CANNOT_CONTINUE condition. The prototype will likely check this value upon entry to every function call. A snazzy implementation may replace all the MPI function calls by function pointers to a dummy function that only returns MPI_ERR_CANNOT_CONTINUE. Resetting these function calls may occur in the error handler at the time original error. This technique would not require a check each time one enters the library. One could imagine other techniques that would equally avoid the overhead of checking this global variable. > >> >> 3) >> I am convinced that the intent of the standard is to require MPI_ERROR_CLASS, MPI_ERROR_STRING and MPI_ABORT to work after an ERRORS_RETURN. If this is insufficiently clear, it should probably be addressed in a stand alone ticket. (it is certainly possible for an error (detected or not) to trash internal state and for that to make one of these three unusable but that applies to every MPI call. The standard does not say MPI_Send must work even if state was scrambled by a wild store). I do not know if anybody assumed MPI_INITIALIZED and MPI_FINALIZED must work after an error. I see no harm in requiring it. > > It is insufficiently clear (to me at at least) from the current standard if these functions are able to be used after an error. During the plenary session last week, it was mentioned by a number of people that this clarification should be part of the formal proposal for this new error class. We can probably pull it out into a separate ticket, but they need to be clarified as part of this ticket anyway. > >> >> 5) >> Finally - I do not see that the ticket does anything useful. In particular, it does not provide any portability improvements I can see. >> >> The MPI implementation could offer a TIMID vs ADVENTUROUS switch (environment variable) >> >> TIMID - MPI query functions like MPI_COMM_SIZE and MPI_ALLOC_MEM do not trigger CANNOT_CONTINUE but every other error does. >> >> ADVENTUROUS - no error triggers CANNOT_CONTINUE. >> >> The default would probably need to be TIMID because if the default were ADVENTUROUS, it would open the implementor to an accusation of failing to protect the customer. There can be no such accusation now because the standard does not imply the implementation should protect the customer. >> >> I have no clue from the ticket what would be a reasonable or portable middle ground. I see the proposal as harmful because any attempt to use it will produce an illusion of portability when implementors try to find a middle ground without guidance form the standard. > > This thread has belabored the point that it is difficult to standardize the middle ground. The expected behavior after an error is returned will likely depend on the error class, MPI call, usage scenario, and, likely, how the call is implemented. For this reason it is difficult to standardize the behavior after any specific error class. If an implementation finds some scenarios where it wishes to be adventurous and continue working then it is allowed to do so, as long as it documents this behavior. > > But what should the MPI implementation do if it finds that it cannot continue working properly any more? It should probably return some type of error to the user letting them know that it has stopped functioning. The current standard says nothing about what should be returned in this scenario. This proposal adds the MPI_ERR_CANNOT_CONTINUE error class to fill this gap. > > With this new error class the user can now know that they should expect MPI_ERR_CANNOT_CONTINUE from all future calls unless the MPI library is attempting to recover. So if they detect a different error class or MPI_SUCCESS then they know the implementation is up to something and to proceed carefully in an implementation specific manner. With the current standard, one implementation may use MPI_ERR_OTHER as the return class for future calls where another implementation may use MPI_ERR_UNKNOWN and still another MPI_ERR_INTERN. So an application is unable to determine if the MPI_ERR_{OTHER|UNKNOWN|INTERN} was because of their use of the interface, or because of the library ceasing to function properly. Instead of mandating that one of the existing error classes fill this role, we introduced the new error class. Now the application can use this as a standardized red flag marking the limits of the ability of the MPI library to function correctly. > > > I acknowledge that this is a small advancement of the standard, but it provides at least some bound on the undefined behavior. So now there is a middle ground to speak of, instead of just everything else after the error. > > -- Josh > > >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> >> From: Joshua Hursey <jjhursey(a)open-mpi.org> >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >> Date: 09/23/2010 08:57 AM >> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >> >> ________________________________ >> >> >> >> (Bringing a lot of points together in a single response) >> >> The ticket that we are discussing is linked below (also part of the very first email in this thread): >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue >> >> < snip > >> >> I deleted the discussion because only the ticket counts now. >> >> >> <ATT00001..txt> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Stabilization Proposal Updated & MPI_COMM_VALIDATE_ALL_SYNC
by Joshua Hursey 29 Sep '10

29 Sep '10

On Sep 28, 2010, at 9:35 AM, Joshua Hursey wrote: > I updated the Run-Through Stabilization proposal: > * Cross reference the MPI_ERR_CANNOT_CONTINUE proposal > * Fix the change for MPI_COMM_SPLIT per a conversation with Jeff Squyres > * Add a MPI_COMM_VALIDATE_ALL_SYNC function (more below) > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > > > MPI_COMM_VALIDATE_ALL_SYNC > --------------------------- > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > > This function will likely replace the existing MPI_COMM_VALIDATE_ALL and MPI_COMM_VALIDATE_ALL_CLEAR as the only collective validation function (and probably be renamed to just MPI_COMM_VALIDATE_ALL). This call is collective over the group/communicator and clears all known failures at the top of the call. It returns a count of the total number of failures (previously recognized and unrecognized) in the group/communicator. The user can use the local MPI_COMM_VALIDATE function to access a list of failures, if needed. > > This function is similar to the original proposal's validate function, but adds the count argument as an agreed upon value. After the discussion in Germany, and experimenting with a few application kernels it became apparent that the MPI_COMM_VALIDATE_ALL and MPI_COMM_VALIDATE_ALL_CLEAR functions are often always called together when a failure happens (incurring two collective calls for each communicator). So, since there is a local accessor to the list, we can create a single collective function that 'fixes' the group/communicator in a single operation. Removing the list of know failures from this collective also reduces the memory footprint needed to call this function. > > A few questions for the group: Per the teleconf discussion: > 1) So are there any objections to removing the MPI_COMM_VALIDATE_ALL and MPI_COMM_VALIDATE_ALL_CLEAR functions and replacing them with a the MPI_COMM_VALIDATE_ALL_SYNC function (and renaming it MPI_COMM_VALIDATE_ALL)? This modification has been made in the proposal. > > 2) Group management operations are local and do not require interprocess communication (Section 6.3). In light of this is there any objection to removing the collective validate functions from the group construction section (they will still be defined for communicators)? The collective group interface has now been removed. Additionally, I renamed the MPI_Rank_state object to MPI_Rank_info since this object represents a bit more than just the state (also includes the 'generation' number which can be seen as part of the rank's name). Cheers, Josh > > > As always, thanks for the feedback. > > -- Josh > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Joshua Hursey 29 Sep '10

29 Sep '10

It was the feeling of the group during the teleconf that this proposal should be suspended for the time being. I drafted a 'proposal resolution' rationale distilled from all of the discussion so far and included it with the proposed text on the wiki. If you feel that something is missing or misrepresented in the resolution text, please let me know. This resolution is this text that we will point the supporters of this idea to when they ask about its status. The text can be found at the following link: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue#Pr… -- Josh On Sep 23, 2010, at 3:54 PM, Joshua Hursey wrote: > > On Sep 23, 2010, at 10:40 AM, Richard Treumann wrote: > >> >> A few quick observations: >> >> 0) >> The constant is MPI_ERROR_ARE_FATAL, not MPI_ERRORS_ABORT > > Fixed. Thanks > >> >> 1) >> The MPI standard only mandates one return code, MPI_SUCCESS. All other return codes are implementation specific and non-portable. For portability, MPI documents error classes and a query function that is passed an implementation defined return code and returns the class. >> >> Assume I allow tags between 0 and 2**15. As an MPI implementor, I am free to use return code 215 for a negative tag and 399 for one that is above 2**15. The error message I print for 215 and the error message I print for 399 can be different. If the user calls MPI_ERROR_CLASS() with either 215 or 399 I give back the class MPI_ARR_TAG. The user who checks the RC of a call to see if it is == MPI_ERR_TAG has written non-portable code. >> >> If I decide return codes 251 and 399 must be in class MPI_CANNOT_CONTINUE they can no longer be in class MPI_ERR_TAG. > > I understand, that is why MPI_ERR_CANNOT_CONTINUE is an error class. We are not defining a new error code, but a new error class. > >> >> >> 2) >> The MPI standard avoids mandating specific error checks. It identifies a lot of errors and in many cases, says what error class that error is in. It does not say an implementation MUST detect the error. I would not violate the standard by skipping the check of whether MPI is initialized. My customers may demand it but the standard does not. You are introducing a mandate for one specific sort of error. > > Yes that is covered in the opening paragraph of 8.3. We are not mandating that an implementation must detect errors, but that if it does detect an error it must return an error code in an error class. So no change in this regard from the existing standard. This proposal adds that if the MPI implementation decides that it cannot continue after it detects the original error then it must return an error code in the MPI_ERR_CANNOT_CONTINUE error class on subsequent MPI calls. So I guess you could say that it must detect that it is itself in an self-defined erroneous state in order to return MPI_ERR_CANNOT_CONTINUE. > > On a related note, the standard language does not specify how an implementation goes about implementing the MPI_ERR_CANNOT_CONTINUE condition. The prototype will likely check this value upon entry to every function call. A snazzy implementation may replace all the MPI function calls by function pointers to a dummy function that only returns MPI_ERR_CANNOT_CONTINUE. Resetting these function calls may occur in the error handler at the time original error. This technique would not require a check each time one enters the library. One could imagine other techniques that would equally avoid the overhead of checking this global variable. > >> >> 3) >> I am convinced that the intent of the standard is to require MPI_ERROR_CLASS, MPI_ERROR_STRING and MPI_ABORT to work after an ERRORS_RETURN. If this is insufficiently clear, it should probably be addressed in a stand alone ticket. (it is certainly possible for an error (detected or not) to trash internal state and for that to make one of these three unusable but that applies to every MPI call. The standard does not say MPI_Send must work even if state was scrambled by a wild store). I do not know if anybody assumed MPI_INITIALIZED and MPI_FINALIZED must work after an error. I see no harm in requiring it. > > It is insufficiently clear (to me at at least) from the current standard if these functions are able to be used after an error. During the plenary session last week, it was mentioned by a number of people that this clarification should be part of the formal proposal for this new error class. We can probably pull it out into a separate ticket, but they need to be clarified as part of this ticket anyway. > >> >> 5) >> Finally - I do not see that the ticket does anything useful. In particular, it does not provide any portability improvements I can see. >> >> The MPI implementation could offer a TIMID vs ADVENTUROUS switch (environment variable) >> >> TIMID - MPI query functions like MPI_COMM_SIZE and MPI_ALLOC_MEM do not trigger CANNOT_CONTINUE but every other error does. >> >> ADVENTUROUS - no error triggers CANNOT_CONTINUE. >> >> The default would probably need to be TIMID because if the default were ADVENTUROUS, it would open the implementor to an accusation of failing to protect the customer. There can be no such accusation now because the standard does not imply the implementation should protect the customer. >> >> I have no clue from the ticket what would be a reasonable or portable middle ground. I see the proposal as harmful because any attempt to use it will produce an illusion of portability when implementors try to find a middle ground without guidance form the standard. > > This thread has belabored the point that it is difficult to standardize the middle ground. The expected behavior after an error is returned will likely depend on the error class, MPI call, usage scenario, and, likely, how the call is implemented. For this reason it is difficult to standardize the behavior after any specific error class. If an implementation finds some scenarios where it wishes to be adventurous and continue working then it is allowed to do so, as long as it documents this behavior. > > But what should the MPI implementation do if it finds that it cannot continue working properly any more? It should probably return some type of error to the user letting them know that it has stopped functioning. The current standard says nothing about what should be returned in this scenario. This proposal adds the MPI_ERR_CANNOT_CONTINUE error class to fill this gap. > > With this new error class the user can now know that they should expect MPI_ERR_CANNOT_CONTINUE from all future calls unless the MPI library is attempting to recover. So if they detect a different error class or MPI_SUCCESS then they know the implementation is up to something and to proceed carefully in an implementation specific manner. With the current standard, one implementation may use MPI_ERR_OTHER as the return class for future calls where another implementation may use MPI_ERR_UNKNOWN and still another MPI_ERR_INTERN. So an application is unable to determine if the MPI_ERR_{OTHER|UNKNOWN|INTERN} was because of their use of the interface, or because of the library ceasing to function properly. Instead of mandating that one of the existing error classes fill this role, we introduced the new error class. Now the application can use this as a standardized red flag marking the limits of the ability of the MPI library to function correctly. > > > I acknowledge that this is a small advancement of the standard, but it provides at least some bound on the undefined behavior. So now there is a middle ground to speak of, instead of just everything else after the error. > > -- Josh > > >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> >> From: Joshua Hursey <jjhursey(a)open-mpi.org> >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >> Date: 09/23/2010 08:57 AM >> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >> >> ________________________________ >> >> >> >> (Bringing a lot of points together in a single response) >> >> The ticket that we are discussing is linked below (also part of the very first email in this thread): >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue >> >> < snip > >> >> I deleted the discussion because only the ticket counts now. >> >> >> <ATT00001..txt> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

[Mpi3-ft] FT Working Group Teleconf
by Joshua Hursey 28 Sep '10

28 Sep '10

Reminder that we have a teleconf tomorrow, Wed 9/29/2010, at 12 noon EDT. Date: Sept. 29, 2010 Time: 12 noon EDT/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Piggybacking (Greg B., George B., Darius B) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - MPI_ERR_CANNOT_CONTINUE Proposal (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue - Run-through stabilization proposal (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - Seek resolution on the MPI_Comm_size issue - Update on progress - other items? Next Meeting: * Oct. 12, 2010 at the MPI Forum meeting in Chicago, IL Respond to the list if you have any additional topics that you would like to discuss. -- Josh ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

[Mpi3-ft] Stabilization Proposal Updated & MPI_COMM_VALIDATE_ALL_SYNC
by Joshua Hursey 28 Sep '10

28 Sep '10

I updated the Run-Through Stabilization proposal: * Cross reference the MPI_ERR_CANNOT_CONTINUE proposal * Fix the change for MPI_COMM_SPLIT per a conversation with Jeff Squyres * Add a MPI_COMM_VALIDATE_ALL_SYNC function (more below) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… MPI_COMM_VALIDATE_ALL_SYNC --------------------------- https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… This function will likely replace the existing MPI_COMM_VALIDATE_ALL and MPI_COMM_VALIDATE_ALL_CLEAR as the only collective validation function (and probably be renamed to just MPI_COMM_VALIDATE_ALL). This call is collective over the group/communicator and clears all known failures at the top of the call. It returns a count of the total number of failures (previously recognized and unrecognized) in the group/communicator. The user can use the local MPI_COMM_VALIDATE function to access a list of failures, if needed. This function is similar to the original proposal's validate function, but adds the count argument as an agreed upon value. After the discussion in Germany, and experimenting with a few application kernels it became apparent that the MPI_COMM_VALIDATE_ALL and MPI_COMM_VALIDATE_ALL_CLEAR functions are often always called together when a failure happens (incurring two collective calls for each communicator). So, since there is a local accessor to the list, we can create a single collective function that 'fixes' the group/communicator in a single operation. Removing the list of know failures from this collective also reduces the memory footprint needed to call this function. A few questions for the group: 1) So are there any objections to removing the MPI_COMM_VALIDATE_ALL and MPI_COMM_VALIDATE_ALL_CLEAR functions and replacing them with a the MPI_COMM_VALIDATE_ALL_SYNC function (and renaming it MPI_COMM_VALIDATE_ALL)? 2) Group management operations are local and do not require interprocess communication (Section 6.3). In light of this is there any objection to removing the collective validate functions from the group construction section (they will still be defined for communicators)? As always, thanks for the feedback. -- Josh ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Joshua Hursey 23 Sep '10

23 Sep '10

On Sep 23, 2010, at 10:40 AM, Richard Treumann wrote: > > A few quick observations: > > 0) > The constant is MPI_ERROR_ARE_FATAL, not MPI_ERRORS_ABORT Fixed. Thanks > > 1) > The MPI standard only mandates one return code, MPI_SUCCESS. All other return codes are implementation specific and non-portable. For portability, MPI documents error classes and a query function that is passed an implementation defined return code and returns the class. > > Assume I allow tags between 0 and 2**15. As an MPI implementor, I am free to use return code 215 for a negative tag and 399 for one that is above 2**15. The error message I print for 215 and the error message I print for 399 can be different. If the user calls MPI_ERROR_CLASS() with either 215 or 399 I give back the class MPI_ARR_TAG. The user who checks the RC of a call to see if it is == MPI_ERR_TAG has written non-portable code. > > If I decide return codes 251 and 399 must be in class MPI_CANNOT_CONTINUE they can no longer be in class MPI_ERR_TAG. I understand, that is why MPI_ERR_CANNOT_CONTINUE is an error class. We are not defining a new error code, but a new error class. > > > 2) > The MPI standard avoids mandating specific error checks. It identifies a lot of errors and in many cases, says what error class that error is in. It does not say an implementation MUST detect the error. I would not violate the standard by skipping the check of whether MPI is initialized. My customers may demand it but the standard does not. You are introducing a mandate for one specific sort of error. Yes that is covered in the opening paragraph of 8.3. We are not mandating that an implementation must detect errors, but that if it does detect an error it must return an error code in an error class. So no change in this regard from the existing standard. This proposal adds that if the MPI implementation decides that it cannot continue after it detects the original error then it must return an error code in the MPI_ERR_CANNOT_CONTINUE error class on subsequent MPI calls. So I guess you could say that it must detect that it is itself in an self-defined erroneous state in order to return MPI_ERR_CANNOT_CONTINUE. On a related note, the standard language does not specify how an implementation goes about implementing the MPI_ERR_CANNOT_CONTINUE condition. The prototype will likely check this value upon entry to every function call. A snazzy implementation may replace all the MPI function calls by function pointers to a dummy function that only returns MPI_ERR_CANNOT_CONTINUE. Resetting these function calls may occur in the error handler at the time original error. This technique would not require a check each time one enters the library. One could imagine other techniques that would equally avoid the overhead of checking this global variable. > > 3) > I am convinced that the intent of the standard is to require MPI_ERROR_CLASS, MPI_ERROR_STRING and MPI_ABORT to work after an ERRORS_RETURN. If this is insufficiently clear, it should probably be addressed in a stand alone ticket. (it is certainly possible for an error (detected or not) to trash internal state and for that to make one of these three unusable but that applies to every MPI call. The standard does not say MPI_Send must work even if state was scrambled by a wild store). I do not know if anybody assumed MPI_INITIALIZED and MPI_FINALIZED must work after an error. I see no harm in requiring it. It is insufficiently clear (to me at at least) from the current standard if these functions are able to be used after an error. During the plenary session last week, it was mentioned by a number of people that this clarification should be part of the formal proposal for this new error class. We can probably pull it out into a separate ticket, but they need to be clarified as part of this ticket anyway. > > 5) > Finally - I do not see that the ticket does anything useful. In particular, it does not provide any portability improvements I can see. > > The MPI implementation could offer a TIMID vs ADVENTUROUS switch (environment variable) > > TIMID - MPI query functions like MPI_COMM_SIZE and MPI_ALLOC_MEM do not trigger CANNOT_CONTINUE but every other error does. > > ADVENTUROUS - no error triggers CANNOT_CONTINUE. > > The default would probably need to be TIMID because if the default were ADVENTUROUS, it would open the implementor to an accusation of failing to protect the customer. There can be no such accusation now because the standard does not imply the implementation should protect the customer. > > I have no clue from the ticket what would be a reasonable or portable middle ground. I see the proposal as harmful because any attempt to use it will produce an illusion of portability when implementors try to find a middle ground without guidance form the standard. This thread has belabored the point that it is difficult to standardize the middle ground. The expected behavior after an error is returned will likely depend on the error class, MPI call, usage scenario, and, likely, how the call is implemented. For this reason it is difficult to standardize the behavior after any specific error class. If an implementation finds some scenarios where it wishes to be adventurous and continue working then it is allowed to do so, as long as it documents this behavior. But what should the MPI implementation do if it finds that it cannot continue working properly any more? It should probably return some type of error to the user letting them know that it has stopped functioning. The current standard says nothing about what should be returned in this scenario. This proposal adds the MPI_ERR_CANNOT_CONTINUE error class to fill this gap. With this new error class the user can now know that they should expect MPI_ERR_CANNOT_CONTINUE from all future calls unless the MPI library is attempting to recover. So if they detect a different error class or MPI_SUCCESS then they know the implementation is up to something and to proceed carefully in an implementation specific manner. With the current standard, one implementation may use MPI_ERR_OTHER as the return class for future calls where another implementation may use MPI_ERR_UNKNOWN and still another MPI_ERR_INTERN. So an application is unable to determine if the MPI_ERR_{OTHER|UNKNOWN|INTERN} was because of their use of the interface, or because of the library ceasing to function properly. Instead of mandating that one of the existing error classes fill this role, we introduced the new error class. Now the application can use this as a standardized red flag marking the limits of the ability of the MPI library to function correctly. I acknowledge that this is a small advancement of the standard, but it provides at least some bound on the undefined behavior. So now there is a middle ground to speak of, instead of just everything else after the error. -- Josh > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Joshua Hursey <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/23/2010 08:57 AM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > ________________________________ > > > > (Bringing a lot of points together in a single response) > > The ticket that we are discussing is linked below (also part of the very first email in this thread): > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue > > < snip > > > I deleted the discussion because only the ticket counts now. > > > <ATT00001..txt> ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 23 Sep '10

23 Sep '10

A few quick observations: 0) The constant is MPI_ERROR_ARE_FATAL, not MPI_ERRORS_ABORT 1) The MPI standard only mandates one return code, MPI_SUCCESS. All other return codes are implementation specific and non-portable. For portability, MPI documents error classes and a query function that is passed an implementation defined return code and returns the class. Assume I allow tags between 0 and 2**15. As an MPI implementor, I am free to use return code 215 for a negative tag and 399 for one that is above 2**15. The error message I print for 215 and the error message I print for 399 can be different. If the user calls MPI_ERROR_CLASS() with either 215 or 399 I give back the class MPI_ARR_TAG. The user who checks the RC of a call to see if it is == MPI_ERR_TAG has written non-portable code. If I decide return codes 251 and 399 must be in class MPI_CANNOT_CONTINUE they can no longer be in class MPI_ERR_TAG. 2) The MPI standard avoids mandating specific error checks. It identifies a lot of errors and in many cases, says what error class that error is in. It does not say an implementation MUST detect the error. I would not violate the standard by skipping the check of whether MPI is initialized. My customers may demand it but the standard does not. You are introducing a mandate for one specific sort of error. 3) I am convinced that the intent of the standard is to require MPI_ERROR_CLASS, MPI_ERROR_STRING and MPI_ABORT to work after an ERRORS_RETURN. If this is insufficiently clear, it should probably be addressed in a stand alone ticket. (it is certainly possible for an error (detected or not) to trash internal state and for that to make one of these three unusable but that applies to every MPI call. The standard does not say MPI_Send must work even if state was scrambled by a wild store). I do not know if anybody assumed MPI_INITIALIZED and MPI_FINALIZED must work after an error. I see no harm in requiring it. 5) Finally - I do not see that the ticket does anything useful. In particular, it does not provide any portability improvements I can see. The MPI implementation could offer a TIMID vs ADVENTUROUS switch (environment variable) TIMID - MPI query functions like MPI_COMM_SIZE and MPI_ALLOC_MEM do not trigger CANNOT_CONTINUE but every other error does. ADVENTUROUS - no error triggers CANNOT_CONTINUE. The default would probably need to be TIMID because if the default were ADVENTUROUS, it would open the implementor to an accusation of failing to protect the customer. There can be no such accusation now because the standard does not imply the implementation should protect the customer. I have no clue from the ticket what would be a reasonable or portable middle ground. I see the proposal as harmful because any attempt to use it will produce an illusion of portability when implementors try to find a middle ground without guidance form the standard. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Joshua Hursey <jjhursey(a)open-mpi.org> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/23/2010 08:57 AM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org (Bringing a lot of points together in a single response) The ticket that we are discussing is linked below (also part of the very first email in this thread): https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue < snip > I deleted the discussion because only the ticket counts now.

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 23 Sep '10

23 Sep '10

Thanks I will read the ticket. I somehow missed the initial pointer and nobody in the follow up discussion ever mentioned it again. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 23 Sep '10

23 Sep '10

Rich Each ticket that comes before the Forum for an up or down vote must stand on its own merits. Discussions cannot change the standard. That requires a ticket to be written and voted. I think no Forum member should vote YES on a ticket unless they think it would be an improvement to the MPI standard EVEN IF every other pending ticket gets defeated. If the concept in the ticket is not an improvement without some broader new feature (like FT), there is no reason to approve it until the rest of the new feature is ready for a vote. At that time it should be folded into the broad proposal that provides the justification. My position at this point is that it is the responsibility of those who want to make a "minor" change to the standard to develop a vote-ready ticket and then it becomes the responsibility of the entire Forum to debate whether that ticket should be approved. I am willing to read and comment on a draft of a ticket before it is entered into the database but I would like to be assured that the authors consider it essentially complete and justified independent of any reference to what may be in the FT chapter. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: "Graham, Richard L." <rlgraham(a)ornl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/23/2010 06:39 AM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org Thanks for the comments - responses in line On 9/23/10 5:45 AM, "Terry Dontje" <terry.dontje(a)oracle.com> wrote: Just saw Dick's email and I guess I will push this rock a little further myself and start attending the FT calls if I can. Graham, Richard L. wrote: Dick, What are your objections here ? All the current proposal is doing is trying to define a set of consistent return codes, and is not changing anything about MPI. I am not sure there is sufficient information to act on in all cases with the current error handling in MPI, but may be wrong on this. However, there is nothing else that is being proposed at this stage. My main objection is that what I heard been discussed does not seem to add value. [rich] Yes, on it's own it adds little value, but not totally useless, but this is not an end of its own. Both you and Richard missed the context of the MPI Forum - not sure why you could not pick this up from the ether :-) Also, if no changes are made to current implementations, there is a chance that apps will hang, but this is no different than if users set errors return today. Which would need to do be done to see the effects of the changes. Now, it is fair to ask if this really adds something. If there is intent to recover from errors, which is what the FT working group is trying to figure out, then this has a lot of value in that it is the venue for an initial discussion of how to extend the error classes, which is really what Josh has been trying to do. I believe we need an implementation and some app experiments to find what we have missed. If this is being proposed just to bootstrap the discussion then fine I think, at least for the group, you've gotten some traction here ;-). :-) As for the choice of "MPI_ERR_CANNOT_CONTINUE" how is this any different than malloc returning null ? The problem as I see it is there might be very few times a library is completely out of commission and probably fewer times than that when the library actually know it is completely out of commission. Now maybe the point here is that this acts as a bridge between the current state of a library that supports MPI 2.2 to when a library supports all of the FT parts of MPI 3.0. That is, as I mentioned before, all libraries that have questions as to there FT support should automatically start returning MPI_ERR_CANNOT_CONTINUE after most/every error until they can fully support FT. [rich] Yes, this is the point. For the sort of environment I work in, the network stack returns sufficient information so that the library can know what is going on. The supporting run-times are also used to detect/notify when process fail, so the library can at least detect state. Responding does involve some changes to the library. I don't think it is unreasonable to respond with a "non-fatal" error code when resource requests can't be satisfied. Ok to be fair, there are probably libraries (OMPI) that do not necessarily handle the loss of a connection very well in certain cases and you could possibly mark a global to say the MPI library is borked in that case. However, if I was going to map out supporting MPI 3.0 FT I'd probably implement the whole thing taking into consideration these cases and returning the appropriate error instead of relying on the MPI_ERR_CANNOT_CONTINUE big hammer. [rich] Not sure that I would mark global state in this manner. But this really depends on how far the implementation wants to go just for this change. I believe the current proposal is a starting point for both the app and the library to start addressing the issues associated with failure. I expect that once you get into the guts of the library, and try to get beyond the equivalent of one big lock for threading, a pile of implementation issues will come up, and this is where the real value of this proposal is. It tells the user that the library is no longer functional, and leaves it to the app to decide how to respond. Depending on the implementation, there are "error" scenarios that both the app and the MPI library can survive. Failure of alloc_mem may be such a function. An app may also decide that sending data to a specific destination may also be ok - I can give a several real use cases that were brought to us as we were looking into this that would be just fine with this. Now the collective operations are another question. So, all this proposal is really doing is start to revive the FT discussion at the Forum level, as partial implementation is getting to a state that it can be evaluated. This is really why it is important to understand the specific shortcomings you see in what is being proposed - just the error propagation issues. Ok, so I see two issues off the bat mentioned: 1. Will an MPI library really know or ever want to throw MPI_ERR_CANNOT_CONTINUE. IMO, I think the answer should be no in that an MPI library should throw errors that are specific to whether a communicator or a connection to a rank is operable. Because I think there are no cases (except for bugs in the library) that a library is completely borked such that nothing can be done. I'll admit today that may not completely be true but in a 3.0 world shouldn't it? [rich] I think that experience will guide us here. I would tend to agree with you, if the library is aiming to recover from failures, but am reluctant to make a sweeping statement w/o more evidence. Libraries that do not want to support recovery are the candidates for returning such error codes. 2. The point Dick makes about the checking of a global value to determine if a library is borked could mess up cache performance is also something we should be concerned with. [rich] Agreed - I would not implement this in that manner. This is ok for some prototyping, but I would tend to implement things in a manner that when failure occurs, the library may do find out about the error deep down in the library, so that the common cause does not take the performance hit. I can actually see how to do this for pt-2-pt communications, but have not thought about collectives, for file ops. Rich --td Thanks, Rich On 9/22/10 6:55 PM, "Richard Treumann" <treumann(a)us.ibm.com> < mailto:[email protected]> wrote: We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. My argument is against adding it into the standard first and figuring out later what it means. I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. Note 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> <mailto:[email protected]> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> <mailto:[email protected]> Date: 09/22/2010 05:47 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org ________________________________ On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: Process can no longer perform any MPI operations. If an MPI operation returns this error class, all subsequent calls to MPI functions will return this error class. Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. -d _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft < http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Joshua Hursey 23 Sep '10

23 Sep '10

(Bringing a lot of points together in a single response) The ticket that we are discussing is linked below (also part of the very first email in this thread): https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue So we are not talking about some ambiguous 'cloud of smoke', but the text as it appears there. If there is some specific language that you have a problem with then we can fix it before it goes before the forum (which is the purpose of this email thread). If you have a problem with the concept of the ticket, then maybe you can advise us on a better rationale scenario to cover in this proposal. As far as performance issues, I am having a really hard time believing that it will impact performance in a substantial way. The vast majority of MPI function calls must check to see if the MPI library has been initialized and not been finalized before calling any lower level functionality. These two checks (and the other parameter checks) add a few branches to the calling path already. If this proposal is to be seen as a big hammer approach, then it adds one more branch statement to the parameter check block to determine if the library can or cannot be used. It is my opinion that the library should already be doing such a check to protect itself from unsafe actions after an error, but the current standard does not require it. As Darius mentioned this is a proposal that defines the upper bound on the space between 'no errors' and 'totally borked'. Where 'no errors' maps to MPI_SUCCESS, and 'totally borked' maps to MPI_ERR_CANNOT_CONTINUE. The MPI implementation can define the behavior in the space between unless the standard mandates some specific behavior in the future. If the implementation only wants to block out some functions (it must document this, as with the current standard) it can use the MPI_ERR_UNSUPPORTED_OPERATION error class. Again, if the MPI implementation does not return MPI_ERR_CANNOT_CONTINUE on all subsequent operations after returning the original error, then it must document this for application/tool developers. In the current standard there is no standard defined return code that serves this purpose, so it is left to the user to hope for the best. With this proposal the application developer will know that if they use an MPI-3 compliant MPI implementation (and this proposal is included) then: if( MPI_SUCCESSS != (ret = MPI_foo()) ) { if( MPI_Is_continuable() ) { /* then we will proceed in an MPI implementation defined manner */ } else { /* All MPI interfaces are blocked from further use */ } } The current standard only allows the 'proceed in an MPI implementation defined manner'. This proposal adds the 'else' case above. This proposal can be seen as a 'big hammer' type of proposal since it requires either MPI_ERR_CANNOT_CONTINUE or MPI implementation defined semantics after an error. The basic implementation will lock the application process receiving the error out of the MPI library after an error. This is a simple check akin to initialized/finalized, with (very) little performance impact. A good quality MPI implementation will (as the proposal preserves from the original standard): -------------- Advice to implementors. A good quality implementation will, to the greatest possible extent, circumscribe the impact of an error, so that normal processing can continue after an error handler was invoked. The implementation documentation will provide information on the possible effect of each class of errors. (End of advice to implementors.) -------------- I would advocate in the implementation that I work with (Open MPI) that we can define sane semantics after returning some error classes under some circumstances. We will document these situations in out user documentation (currently the FAQ), and if enough users like these semantics we might even build up our courage to propose them before the fault tolerance working group then the full forum. For all those scenarios where we cannot decide or determine sane semantics after an error class, we need something that we can return to the user letting them know that we do not want to proceed. This proposal introduces the MPI_ERR_CANNOT_CONTINUE error class to serve this purpose. This conversation has escalated into a heated debate, which is great since I would rather have this type of discussion in the working group context instead of the full forum context. This debate will hopefully make the proposal better (or possibly even kill it). I appreciate everyones feedback and patience with this thread. Needless to say this proposal will be a topic for discussion in our next teleconf (on Sept. 29). Hopefully a phone conversation will help us break through some of the confusion. In the mean time we should continue to discuss on the mailing list and hopefully work towards some resolution/clarification. Cheers, Josh On Sep 23, 2010, at 6:37 AM, Graham, Richard L. wrote: > Thanks for the comments - responses in line > > > On 9/23/10 5:45 AM, "Terry Dontje" <terry.dontje(a)oracle.com> wrote: > > Just saw Dick's email and I guess I will push this rock a little further myself and start attending the FT calls if I can. > > Graham, Richard L. wrote: > > Dick, > What are your objections here ? All the current proposal is doing is trying to define a set of consistent return codes, and is not changing anything about MPI. I am not sure there is sufficient information to act on in all cases with the current error handling in MPI, but may be wrong on this. However, there is nothing else that is being proposed at this stage. > > My main objection is that what I heard been discussed does not seem to add value. > > [rich] Yes, on it's own it adds little value, but not totally useless, but this is not an end of its own. Both you and Richard missed the context of the MPI Forum - not sure why you could not pick this up from the ether :-) > > Also, if no changes are made to current implementations, there is a chance that apps will hang, but this is no different than if users set errors return today. Which would need to do be done to see the effects of the changes. > Now, it is fair to ask if this really adds something. If there is intent to recover from errors, which is what the FT working group is trying to figure out, then this has a lot of value in that it is the venue for an initial discussion of how to extend the error classes, which is really what Josh has been trying to do. I believe we need an implementation and some app experiments to find what we have missed. > > If this is being proposed just to bootstrap the discussion then fine I think, at least for the group, you've gotten some traction here ;-). > > :-) > > As for the choice of "MPI_ERR_CANNOT_CONTINUE" how is this any different than malloc returning null ? > The problem as I see it is there might be very few times a library is completely out of commission and probably fewer times than that when the library actually know it is completely out of commission. Now maybe the point here is that this acts as a bridge between the current state of a library that supports MPI 2.2 to when a library supports all of the FT parts of MPI 3.0. That is, as I mentioned before, all libraries that have questions as to there FT support should automatically start returning MPI_ERR_CANNOT_CONTINUE after most/every error until they can fully support FT. > > [rich] Yes, this is the point. For the sort of environment I work in, the network stack returns sufficient information so that the library can know what is going on. The supporting run-times are also used to detect/notify when process fail, so the library can at least detect state. Responding does involve some changes to the library. I don't think it is unreasonable to respond with a "non-fatal" error code when resource requests can't be satisfied. > > Ok to be fair, there are probably libraries (OMPI) that do not necessarily handle the loss of a connection very well in certain cases and you could possibly mark a global to say the MPI library is borked in that case. However, if I was going to map out supporting MPI 3.0 FT I'd probably implement the whole thing taking into consideration these cases and returning the appropriate error instead of relying on the MPI_ERR_CANNOT_CONTINUE big hammer. > > [rich] Not sure that I would mark global state in this manner. But this really depends on how far the implementation wants to go just for this change. I believe the current proposal is a starting point for both the app and the library to start addressing the issues associated with failure. I expect that once you get into the guts of the library, and try to get beyond the equivalent of one big lock for threading, a pile of implementation issues will come up, and this is where the real value of this proposal is. > > It tells the user that the library is no longer functional, and leaves it to the app to decide how to respond. Depending on the implementation, there are "error" scenarios that both the app and the MPI library can survive. Failure of alloc_mem may be such a function. An app may also decide that sending data to a specific destination may also be ok - I can give a several real use cases that were brought to us as we were looking into this that would be just fine with this. Now the collective operations are another question. > So, all this proposal is really doing is start to revive the FT discussion at the Forum level, as partial implementation is getting to a state that it can be evaluated. This is really why it is important to understand the specific shortcomings you see in what is being proposed - just the error propagation issues. > > > Ok, so I see two issues off the bat mentioned: > > 1. Will an MPI library really know or ever want to throw MPI_ERR_CANNOT_CONTINUE. > IMO, I think the answer should be no in that an MPI library should throw errors that are > specific to whether a communicator or a connection to a rank is operable. Because I think > there are no cases (except for bugs in the library) that a library is completely borked > such that nothing can be done. I'll admit today that may not completely be true but in a 3.0 > world shouldn't it? > > [rich] I think that experience will guide us here. I would tend to agree with you, if the library is aiming to recover from failures, but am reluctant to make a sweeping statement w/o more evidence. Libraries that do not want to support recovery are the candidates for returning such error codes. > > 2. The point Dick makes about the checking of a global value to determine if a library > is borked could mess up cache performance is also something we should be concerned with. > [rich] Agreed - I would not implement this in that manner. This is ok for some prototyping, but I would tend to implement things in a manner that when failure occurs, the library may do find out about the error deep down in the library, so that the common cause does not take the performance hit. I can actually see how to do this for pt-2-pt communications, but have not thought about collectives, for file ops. > > Rich > > --td > > Thanks, > Rich > > On 9/22/10 6:55 PM, "Richard Treumann" <treumann(a)us.ibm.com> <mailto:[email protected]> wrote: > > > We are kind of going in circles because the context and rationale for CANNOT_CONTINUE are still too ambiguous. > > My argument is against adding it into the standard first and figuring out later what it means. > > I will wait for the ticket. If the ticket gives a full and convincing specification of what the implementor and the user are to do with it,, I will make my judgement based on the whole description. > > If the ticket says "Put this minor change in today and we will decide later what it means, I must lobby the Forum to reject the ticket.. > > Note > 1) all current errors detected by an MPI application map to an existing error class. An error cannot map to two error classes so if some user error handler is presently checking for MPI_ERR_OP after a non-SUCCESS return from MPI_Reduce and the implementation moves the return code for passing a bad OP from class MPI_ERR_OP to MPI_ERR_CANNOT_CONTINUE it has just broken a user code. > 2) Mandating that every MPI call after a MPI_ERR_CANNOT_CONTINUE must return MPI_ERR_CANNOT_CONTINUE will require that every MPI call check a global flag (resulting in overhead and possible displacement of other data from cache) > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Darius Buntinas <buntinas(a)mcs.anl.gov> <mailto:[email protected]> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> <mailto:[email protected]> > Date: 09/22/2010 05:47 PM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > ________________________________ > > > > > On Sep 22, 2010, at 2:29 PM, Richard Treumann wrote: > > > > > You lost me there - in part, i am saying it is useless because there are almost zero cases in which it would be appropriate. How does that make it "a minor change"? > > > > > Well I figure we're just adding an error class that the implementation can return to the user if it gives up and can't continue. That's minor. Whether or not it's useful is another story :-) > > > > > Can you provide me the precise text you would add to the standard? Exactly how does the CANNOT_CONTINUE work? Under what conditions does an MPI process see a CANNOT_CONTINUE and what does it mean? > > > > > I don't know yet. It might be something as simple as adding an entry to the error class table with a description like: > > Process can no longer perform any MPI operations. If an MPI operation > returns this error class, all subsequent calls to MPI functions will > return this error class. > > > > > Please look at the example again. The point was that there is nothing there that would justify a CANNOT_CONTINUE and MPI is still working correctly. Despite that, the behavior is a mess from the algorithm viewpoint after the error. > > > > > Since we haven't defined what happens in a failed collective yet, consider an implementation could will not continue after a failed collective. The odd numbered processes that did not immediately return from barrier with an error will continue with the barrier protocol (say it's recursive doubling). Some of the odd processes will need to send messages to some of the even processes. Upon receiving these messages, the even processes will respond with an I_QUIT message, or perhaps the connection is closed, so the odd processes will get a communication error when trying to send the message. In either case, the odd processes will notice that something's wrong with the other processes, and return an error. The second barrier will return a CANNOT_CONTINUE on all of the processes. > > OK, what if the odd processes can't determine that the even processes can't continue? The odd processes would hang in the first barrier, and the even numbered processes would get a CANNOT_CONTINUE from the second barrier. > > So we either get a hang, or everyone gets a CANNOT_CONTINUE but we avoided the discombobulated scenario. > > -d > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0