September 2010 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question
by Graham, Richard L. 20 Sep '10

20 Sep '10

My objection is not defining what the value of mpi_comm_size() returns. I think that it should return a well define value, and I would advocate for the number of ranks that have been started, which is what it returns today. If we want to be able to have users ask about command line options - soft limits, hard limits, or ?, we should provide a standard way to ask this question. I may have missed what you are proposing, but "pick your meaning of a return value" seems like it would lead to non-portable apps. Rich On 9/20/10 10:51 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: How would it be specified? "To start an MPI application the user should call "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or will fail with error code XXX." This would make any runtime that provides the -soft parameter noncompliant. Do we really want to do this? Alternatively, we could define it something like: "To start an MPI application the user should cal "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or another number or MPI processes depending on other parameters specified by the user, or fail with an error code." But isn't this what we have now? What's wrong with that? -d On Sep 20, 2010, at 9:39 AM, Graham, Richard L. wrote: > I strongly disagree here. This is such a basic bit of data, we clearly need to specify the output. > > Rich > > ----- Original Message ----- > From: Joshua Hursey [mailto:[email protected]] > Sent: Monday, September 20, 2010 10:09 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group <mpi3-ft(a)lists.mpi-forum.org> > Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question > > This is an interesting discussion (I am glad it is archived). > > I feel that Darius makes a strong argument about leaving it undefined in the standard for many of the reasons that have been mentioned. I think an implementation can decide then if one way is more important to their users than another. It is my opinion that defining it does not gain us much since most applications base their initialization on the result of MPI_Comm_size() instead of requiring a specific number of processes. Further, if they do require a specific number, they can interpret the results of MPI_Comm_size() and choose to abort or work around the difference. > > -- Josh > > On Sep 17, 2010, at 11:00 AM, Graham, Richard L. wrote: > >> Why would we select one behaviour over another ? Right now users use the value returned from mpi_comm_size() to determine how to setup their parallel job, so telling them what the actually got seems like the most obvious return value to have, especially if we do not provide a way to get both what they asked for, and what they actually got. Having this be implementation specific does not seem to be prudent, since they would have to know what implementation they are using to figure out what to do with the data. I believe this is very different than an implementation choice to use an eager protocol for large message vs. a rendezvous which does not have correctness implications. >> >> Also, at startup the user has not set anything up yet, with respect to MPI, so forming a dense communicator is well defined and the easiest thing to do. Once failure occurs, I will argue exactly the opposite, based on the outcome of such a decisions, and the fact that we will provide them a way within MPI to figure out what has happened. >> >> Rich >> >> On 9/17/10 10:14 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: >> >> >> >> I don't think we need to choose one or the other (in fact I feel strongly that we should not force one behavior or the other). The choice to have MPI_COMM_WORLD contain N or M processes (or failing if it can't get all N) is implementation dependent. Presumably the behavior would be user selectable (e.g., using a -soft option or something similar to mpiexec). >> >> The user would use the mechanisms we will provide to deal with any dead processes (e.g., validating a communicator, etc). >> >> -d >> >> On Sep 17, 2010, at 4:02 PM, Bronevetsky, Greg wrote: >> >>> I agree that MPI_Comm_size should return the number of ranks in the communicator regardless of whether they're operational or not. However, this just pushes the question further back: if the user asked for N processes but only M have started up, how many ranks should MPI_COMM_WORLD have? Either choice is going to be self-consistent from MPI's point of view. If its N, then some ranks will be dead. If it is M then the application may not have enough processes to work with. The former (N) case has property that the user doesn't need to add code to check for this condition since their existing error checking code will catch this situation. The latter case (M) is nice because it cheaper to check whether we got fewer processes than expected than to explicitly try to communicate with them. >>> >>> As such, I don't see a strong motivation for choosing either. However, we should just pick one and stick with it to avoid unnecessary API divergence. >>> >>> Greg Bronevetsky >>> Lawrence Livermore National Lab >>> (925) 424-5756 >>> bronevetsky(a)llnl.gov >>> http://greg.bronevetsky.com >>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>> bounces(a)lists.mpi-forum.org] On Behalf Of Bronis R. de Supinski >>>> Sent: Friday, September 17, 2010 5:23 AM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >>>> >>>> >>>> I agree with Rich and Darius. >>>> >>>> On Fri, 17 Sep 2010, Darius Buntinas wrote: >>>> >>>>> >>>>> I don't think we should change the standard in this case. For >>>> MPI_Comm_size to have any useful meaning, it needs to return the size of >>>> the communicator: i.e., if comm_size returns N, you should be able to do a >>>> send to processes 0 through N-1. Of course if some of those processes have >>>> failed, you'll get an error associated with the process failure, but never >>>> an error for an invalid rank. >>>>> >>>>> As discussed in the section about mpiexec, an implementation may decide >>>> to provide a soft process count argument. So "mpiexec -n 10 -soft 5 ./cpi" >>>> can start any number between 5 and 10 processes. But that does not affect >>>> the meaning of the size of MPI_COMM_WORLD: regardless of the number of >>>> processes the implementation decides to start, MPI_Comm_size will return >>>> the _actual_ number of processes started. >>>>> >>>>> -d >>>>> >>>>> On Sep 17, 2010, at 11:22 AM, Graham, Richard L. wrote: >>>>> >>>>>> We need to clearly define what N or M is and not leave it to the >>>> implementation. 100% of the codes that seen over the past 15 years that >>>> check this value use it to indicate how many process have started. Any >>>> thing else is really useless, aside from letting the user find out how many >>>> processes actually started up, and then know how many did not start up. >>>>>> >>>>>> Rich >>>>>> >>>>>> >>>>>> On 9/17/10 4:27 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: >>>>>> >>>>>> So the Run-Through Stabilization proposal has been updated per our >>>> discussion in the working group meeting at the MPI Forum. The changes are >>>> summarized below: >>>>>> - Add a Legacy Library Support example >>>>>> - Clarify new error classes >>>>>> - Update the MPI_Init and MPI_Finalize wording to be simpler and more >>>> direct. >>>>>> - Fix wording of group creation calls versus communicator creation >>>> calls. >>>>>> >>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>> web/wiki/ft/run_through_stabilization >>>>>> >>>>>> >>>>>> One question that we discussed quite a bit during the meeting was the >>>> issue of the return value of MPI_Comm_size() when processes fail during >>>> launch. I attempted to capture the discussion in the room in the Open >>>> Question attached to the discussion of MPI_Init: >>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>> web/wiki/ft/run_through_stabilization#MPI_INIT >>>>>> >>>>>> Open question: >>>>>> If the user asks to start N processes on the command line, and only M >>>> processes were successfully launched (where M < N), then what should be >>>> returned from MPI_COMM_SIZE? >>>>>> >>>>>> The return value must be consistent across all alive members of the >>>> group. The issue is if it should return N or M. >>>>>> >>>>>> The feeling in the room was that since the MPI standard does not define >>>> the ability for the user to ask for a specific number of processes before >>>> initthen it is hard to define that this is the number it should be. >>>>>> >>>>>> So it is left to the implementation whether it is M or N. If it is M, >>>> then the user has other techniques to find out what it originally asked for >>>> (e.g., by passing it as a command line argument to the application itself). >>>>>> >>>>>> >>>>>> What do people think about the MPI_Comm_size issue? >>>>>> >>>>>> -- Josh >>>>>> >>>>>> ------------------------------------ >>>>>> Joshua Hursey >>>>>> Postdoctoral Research Associate >>>>>> Oak Ridge National Laboratory >>>>>> http://BLOCKEDBLOCKEDwww.BLOCKEDBLOCKEDcs.indiana.edu/~jjhursey >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Joshua Hursey 20 Sep '10

20 Sep '10

So the proposal makes no requirements about the state of the distributed environment after an error. All it defines is a error class to be returned by an MPI implementation once the MPI implementation can no longer function correctly. This provides a necessary feedback mechanism for the application to determine if and how the MPI implementation can be used after an error occurs. It is the responsibility of the application to avoid deadlocks and other such issues that can result from handling and recovering from errors. If an application is designed to recover from MPI_ERR_TAG that's great. If the MPI library allows for correct operation after returning that error class, then even better. If the MPI library cannot continue operation after that error then it can block subsequent operations by returning MPI_ERR_CANNOT_CONTINUE. I disagree with your assessment that this will be difficult to implement/test since a trivial implementation of this proposal is to set a global variable when an error occurs to always return MPI_ERR_CANNOT_CONTINUE when the application calls into the MPI library (it is a similar check as the 'is_mpi_initialized' check that has to be there anyway). If an implementation wants to do more (and it is definitely not required to do so) then it can define that in it's documentation. If an application wants to try to use MPI after an error it must understand that the error is local in nature (it cannot assume that every process received an error). If it can figure out how to recover from it, and the MPI implementation is able to function correctly afterward then we should let them figure it out. This allows us to define the boundaries of correct operation after an error. Before the application -could- keep using the MPI library after an error, but it was entirely undefined and not-portable what would happen. Now the application can portably attempt to use the MPI library after an error and know that it can expect either normal functionality (for those few implementations that do more than the minimum necessary) or MPI_ERR_CANNOT_CONTINUE in which the library locks them out and they then know to terminate normally. I hope this helps a bit, but maybe I am missing the core problem that you are trying to get at. -- Josh On Sep 20, 2010, at 1:09 PM, Bronis R. de Supinski wrote: > > Dick: > > Re: >> I did not intend to ignore your use case. > > No problem. > >> I did mention that I have no worries about asking MPI implementations >> to refrain from blocking future MPI calls after an error is detected. >> That was an implicit recognition of your use case. > > OK, that helps. > >> The MPI standard already forbids having an MPI call on one thread block >> progress on other threads. I would interpret that to include a case >> where a thread is blocked in a collective communication or a MPI_Recv >> that will never be satisfied. That is, the blocked MPI call cannot >> prevent other threads from using libmpi. Requiring libmpi to release >> any lock it took even when doing an error return would be logical but >> may not be implied by what is currently written. > > The current text provides no such guarantee. Once anerror is > returned anywhere, all bets are off (at least that is how I > have read it; I would need to go back through the text to > find the exact words that cause my concern). > >> Communicators provide a sort of isolation that keeps stray crap from >> failed operations from spilling over (such as eager sent message for >> which the MPI_Recv failed). If the tool uses its own threads and >> private communicators, I agree it is reasonable to ask any libmpi to >> avoid sabotaging that communication. > > That would be perfect from my perspective. > >> Where I get concerned is when we start talking about affirmative >> requirements for distributed MPI state after an error > > I don't think we can have those beyond "best effort". > The errors may indicate problems that make further > communication impossible -- perhaps because of the > erroneous action or just due to the state of the > network or other processes. I do think we can require > accurate return values and have an advice to implementers > that suggests best effort following errors. I believe > that would satisfy my requirements. > > Bronis > > > > >> >> Dick >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> >> From: "Bronis R. de Supinski" <bronis(a)llnl.gov> >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >> Date: 09/20/2010 12:46 PM >> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >> >> ________________________________ >> >> >> >> >> Dick: >> >> You seem to be ignoring my use case. Specifically, I >> have tool threads that use MPI. Their use of MPI should >> be unaffected by all of the scenarios that you are raising. >> However, the standard provides no way for me to tell if >> they work correctly in these situations. I just have to >> cross my fingers and hope. >> >> FYI: Your implementation has long met this requirement >> (my hopes are not dashed with it). Others have begun to >> recently. In any event, I would like some way to tell... >> >> Further, it is useful in many other scenarios apply to know >> that the implementation intends to remain usable. I am not >> looking for a promise of correct execution; I am looking >> for a promise of best effort and accurate return codes. >> >> Bronis >> >> >> >> On Mon, 20 Sep 2010, Richard Treumann wrote: >> >>> >>> If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) >>> >>> MPI_Abort, >>> MPI_Error_string >>> MPI_Error_class >>> >>> I assume it should be corrected as a trivial oversight in the original text. >>> >>> I would regard the real issue as being the difficulty with assuring the state of remote processes. >>> >>> There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. >>> >>> For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. >>> >>> >>> >>> >>> >>> >>> Dick Treumann - MPI Team >>> IBM Systems & Technology Group >>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>> Tele (845) 433-7846 Fax (845) 433-8363 >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Bronis R. de Supinski 20 Sep '10

20 Sep '10

Dick: Re: > I did not intend to ignore your use case. No problem. > I did mention that I have no worries about asking MPI implementations > to refrain from blocking future MPI calls after an error is detected. > That was an implicit recognition of your use case. OK, that helps. > The MPI standard already forbids having an MPI call on one thread block > progress on other threads. I would interpret that to include a case > where a thread is blocked in a collective communication or a MPI_Recv > that will never be satisfied. That is, the blocked MPI call cannot > prevent other threads from using libmpi. Requiring libmpi to release > any lock it took even when doing an error return would be logical but > may not be implied by what is currently written. The current text provides no such guarantee. Once anerror is returned anywhere, all bets are off (at least that is how I have read it; I would need to go back through the text to find the exact words that cause my concern). > Communicators provide a sort of isolation that keeps stray crap from > failed operations from spilling over (such as eager sent message for > which the MPI_Recv failed). If the tool uses its own threads and > private communicators, I agree it is reasonable to ask any libmpi to > avoid sabotaging that communication. That would be perfect from my perspective. > Where I get concerned is when we start talking about affirmative > requirements for distributed MPI state after an error I don't think we can have those beyond "best effort". The errors may indicate problems that make further communication impossible -- perhaps because of the erroneous action or just due to the state of the network or other processes. I do think we can require accurate return values and have an advice to implementers that suggests best effort following errors. I believe that would satisfy my requirements. Bronis > > Dick > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: "Bronis R. de Supinski" <bronis(a)llnl.gov> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/20/2010 12:46 PM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > ________________________________ > > > > > Dick: > > You seem to be ignoring my use case. Specifically, I > have tool threads that use MPI. Their use of MPI should > be unaffected by all of the scenarios that you are raising. > However, the standard provides no way for me to tell if > they work correctly in these situations. I just have to > cross my fingers and hope. > > FYI: Your implementation has long met this requirement > (my hopes are not dashed with it). Others have begun to > recently. In any event, I would like some way to tell... > > Further, it is useful in many other scenarios apply to know > that the implementation intends to remain usable. I am not > looking for a promise of correct execution; I am looking > for a promise of best effort and accurate return codes. > > Bronis > > > > On Mon, 20 Sep 2010, Richard Treumann wrote: > >> >> If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) >> >> MPI_Abort, >> MPI_Error_string >> MPI_Error_class >> >> I assume it should be corrected as a trivial oversight in the original text. >> >> I would regard the real issue as being the difficulty with assuring the state of remote processes. >> >> There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. >> >> For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. >> >> >> >> >> >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 20 Sep '10

20 Sep '10

Sorry Bronis - I did not intend to ignore your use case. I did mention that I have no worries about asking MPI implementations to refrain from blocking future MPI calls after an error is detected. That was an implicit recognition of your use case. The MPI standard already forbids having an MPI call on one thread block progress on other threads. I would interpret that to include a case where a thread is blocked in a collective communication or a MPI_Recv that will never be satisfied. That is, the blocked MPI call cannot prevent other threads from using libmpi. Requiring libmpi to release any lock it took even when doing an error return would be logical but may not be implied by what is currently written. Communicators provide a sort of isolation that keeps stray crap from failed operations from spilling over (such as eager sent message for which the MPI_Recv failed). If the tool uses its own threads and private communicators, I agree it is reasonable to ask any libmpi to avoid sabotaging that communication. Where I get concerned is when we start talking about affirmative requirements for distributed MPI state after an error Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: "Bronis R. de Supinski" <bronis(a)llnl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/20/2010 12:46 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org Dick: You seem to be ignoring my use case. Specifically, I have tool threads that use MPI. Their use of MPI should be unaffected by all of the scenarios that you are raising. However, the standard provides no way for me to tell if they work correctly in these situations. I just have to cross my fingers and hope. FYI: Your implementation has long met this requirement (my hopes are not dashed with it). Others have begun to recently. In any event, I would like some way to tell... Further, it is useful in many other scenarios apply to know that the implementation intends to remain usable. I am not looking for a promise of correct execution; I am looking for a promise of best effort and accurate return codes. Bronis On Mon, 20 Sep 2010, Richard Treumann wrote: > > If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) > > MPI_Abort, > MPI_Error_string > MPI_Error_class > > I assume it should be corrected as a trivial oversight in the original text. > > I would regard the real issue as being the difficulty with assuring the state of remote processes. > > There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. > > For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. > > > > > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Bronis R. de Supinski 20 Sep '10

20 Sep '10

Dick: You seem to be ignoring my use case. Specifically, I have tool threads that use MPI. Their use of MPI should be unaffected by all of the scenarios that you are raising. However, the standard provides no way for me to tell if they work correctly in these situations. I just have to cross my fingers and hope. FYI: Your implementation has long met this requirement (my hopes are not dashed with it). Others have begun to recently. In any event, I would like some way to tell... Further, it is useful in many other scenarios apply to know that the implementation intends to remain usable. I am not looking for a promise of correct execution; I am looking for a promise of best effort and accurate return codes. Bronis On Mon, 20 Sep 2010, Richard Treumann wrote: > > If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) > > MPI_Abort, > MPI_Error_string > MPI_Error_class > > I assume it should be corrected as a trivial oversight in the original text. > > I would regard the real issue as being the difficulty with assuring the state of remote processes. > > There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. > > For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. > > > > > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 >

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 20 Sep '10

20 Sep '10

I realize my basic point is rather simple: If you say MPI state is UNDEFINED after an error, you have part of the standard done. Simple and clear. If you say MPI state is DEFINED after certain errors, the standard must define it. The complexity of identifying and classifying errors and defining the state after each is mind boggling. The job of implementing and testing that is even more mind boggling. BTW - I see no practical problem with mandating that an error code in class NOT_IMPLEMENTED be considered harmless. I.E. "A subroutine that returns an error in error class NOT_IMPLEMENTED leaves the MPI library in the same state as if the call had not been made." That is easy to define and implement. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 20 Sep '10

20 Sep '10

If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) MPI_Abort, MPI_Error_string MPI_Error_class I assume it should be corrected as a trivial oversight in the original text. I would regard the real issue as being the difficulty with assuring the state of remote processes. There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Darius Buntinas 20 Sep '10

20 Sep '10

Currently the behavior is undefined, so the implementation could really do anything...including the proposed behavior, so this shouldn't affect backward compatibility. Note that this doesn't affect the ERRORS_ARE_FATAL case, just the ERRORS_RETURN case, so apps that don't set an error handler and count on the implementation aborting, will not be affected. -d On Sep 20, 2010, at 11:00 AM, Richard Treumann wrote: > > I am not talking about libmpi fixing an application bug. I am talking about the fact that if an application has a bug, the state of the application becomes unknown. Something that was part of the algorithm that the author was trying to apply to get an answer has not happened as envisioned. How can the application state be trusted? > > I see no problem with urging MPI implementations to refrain from shooting down future MPI calls when the user has set MPI_ERRORS_RETURN but I have a hard time imagining going much beyond that for application bugs. > > For example, a call to MPI_Bcast that has a bad communicator at one task will eventually hang but one that has a bad communicator at all tasks can continue (the application state is probably corrupted but libmpi should be OK) > > > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Darius Buntinas <buntinas(a)mcs.anl.gov> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/20/2010 10:43 AM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > > > > > I don't think Josh meant that the MPI implementation would fix application bugs, but rather that the return of an error class other than CANNOT_CONTINUE means that the implementation is in an internally consistent state and that it can continue performing MPI functions. > > -d > > On Sep 20, 2010, at 9:33 AM, Richard Treumann wrote: > > > > > How does an application experience errors in classes (MPI_ERR_COUNT, MPI_ERR_TAG) except by a bug in the application itself? > > > > How can it be easier for someone to know how to continue from an arbitrary application bug with confidence that the application is still giving good answers, than to just fix the app? > > > > > > Dick Treumann - MPI Team > > IBM Systems & Technology Group > > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > > Tele (845) 433-7846 Fax (845) 433-8363 > > > > > > > > From: Joshua Hursey <jjhursey(a)open-mpi.org> > > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > > Date: 09/20/2010 10:05 AM > > Subject: [Mpi3-ft] Defining the state of MPI after an error > > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > > > > > > > > > During EuroMPI and the MPI Forum meeting last week the issue of the MPI state after an error was brought up a few times. The issue is that since the state is undefined then no portable program can be written that uses the errorhandlers then MPI functionality following the error. This issue is particularly difficult for applications that wish to catch informational or warning type errors (e.g., MPI_ERR_COUNT, MPI_ERR_TAG, MPI_ERR_UNSUPPORTED_OPERATION). These operations are often recoverable by the MPI implementation and/or the application. > > > > To address this portability issue, I am bringing out the MPI_ERR_CANNOT_CONTINUE error class from the stabilization proposal. I presented the idea to the MPI Forum during a plenary session last week and received a positive response on building a formal proposal [Straw vote: 22 (yes), 0 (no), 3 (abstain)]. > > > > I have created a first draft of the proposal for the working group to review on the wiki at the link below: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue > > > > I would like to have this proposal ready by the Oct. meeting so we can have a formal plenary session on it. If all goes well, maybe we can get a first reading by Dec. > > > > Let me know what you think about this proposal. > > > > -- Josh > > > > ------------------------------------ > > Joshua Hursey > > Postdoctoral Research Associate > > Oak Ridge National Laboratory > > http://www.cs.indiana.edu/~jjhursey > > > > > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 20 Sep '10

20 Sep '10

I am not talking about libmpi fixing an application bug. I am talking about the fact that if an application has a bug, the state of the application becomes unknown. Something that was part of the algorithm that the author was trying to apply to get an answer has not happened as envisioned. How can the application state be trusted? I see no problem with urging MPI implementations to refrain from shooting down future MPI calls when the user has set MPI_ERRORS_RETURN but I have a hard time imagining going much beyond that for application bugs. For example, a call to MPI_Bcast that has a bad communicator at one task will eventually hang but one that has a bad communicator at all tasks can continue (the application state is probably corrupted but libmpi should be OK) Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Darius Buntinas <buntinas(a)mcs.anl.gov> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/20/2010 10:43 AM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org I don't think Josh meant that the MPI implementation would fix application bugs, but rather that the return of an error class other than CANNOT_CONTINUE means that the implementation is in an internally consistent state and that it can continue performing MPI functions. -d On Sep 20, 2010, at 9:33 AM, Richard Treumann wrote: > > How does an application experience errors in classes (MPI_ERR_COUNT, MPI_ERR_TAG) except by a bug in the application itself? > > How can it be easier for someone to know how to continue from an arbitrary application bug with confidence that the application is still giving good answers, than to just fix the app? > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Joshua Hursey <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/20/2010 10:05 AM > Subject: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > > > > During EuroMPI and the MPI Forum meeting last week the issue of the MPI state after an error was brought up a few times. The issue is that since the state is undefined then no portable program can be written that uses the errorhandlers then MPI functionality following the error. This issue is particularly difficult for applications that wish to catch informational or warning type errors (e.g., MPI_ERR_COUNT, MPI_ERR_TAG, MPI_ERR_UNSUPPORTED_OPERATION). These operations are often recoverable by the MPI implementation and/or the application. > > To address this portability issue, I am bringing out the MPI_ERR_CANNOT_CONTINUE error class from the stabilization proposal. I presented the idea to the MPI Forum during a plenary session last week and received a positive response on building a formal proposal [Straw vote: 22 (yes), 0 (no), 3 (abstain)]. > > I have created a first draft of the proposal for the working group to review on the wiki at the link below: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue > > I would like to have this proposal ready by the Oct. meeting so we can have a formal plenary session on it. If all goes well, maybe we can get a first reading by Dec. > > Let me know what you think about this proposal. > > -- Josh > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Joshua Hursey 20 Sep '10

20 Sep '10

Yeah. So we are just defining what MPI should do if the application tried to call back into it after an error. We are not assessing if the application is erroneous by receiving the error in the first place. Certainly for MPI_ERR_UNSUPPORTED_OPERATION it is not the application's fault that the MPI library does not support the operation, and the application may very well be able to work around unsupported operations. For other error classes, it is slightly less clear if it is an application bug or not. Since the state of MPI is undefined after an error, it is possible that the user could call an MPI function after receiving an error and MPI returns SUCCESS even though it may not have done anything that the user expected. Additionally, it is undefined if the application can call MPI_ERROR_STRING to get a string to display before calling MPI_Abort (also undefined if it can be called). So this proposal just defines an ability for the MPI implementation to lock the user out of the MPI library if it can no longer continue operating normally. This way the application can look for the MPI_ERR_CANNOT_CONTINUE error class to know when the MPI library is no longer usable. Further, if it receives any other return code other than MPI_ERR_CANNOT_CONTINUE, it knows that the MPI library is behaving normally for that operation. -- Josh On Sep 20, 2010, at 10:42 AM, Darius Buntinas wrote: > > I don't think Josh meant that the MPI implementation would fix application bugs, but rather that the return of an error class other than CANNOT_CONTINUE means that the implementation is in an internally consistent state and that it can continue performing MPI functions. > > -d > > On Sep 20, 2010, at 9:33 AM, Richard Treumann wrote: > >> >> How does an application experience errors in classes (MPI_ERR_COUNT, MPI_ERR_TAG) except by a bug in the application itself? >> >> How can it be easier for someone to know how to continue from an arbitrary application bug with confidence that the application is still giving good answers, than to just fix the app? >> >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> >> From: Joshua Hursey <jjhursey(a)open-mpi.org> >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >> Date: 09/20/2010 10:05 AM >> Subject: [Mpi3-ft] Defining the state of MPI after an error >> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >> >> >> >> >> During EuroMPI and the MPI Forum meeting last week the issue of the MPI state after an error was brought up a few times. The issue is that since the state is undefined then no portable program can be written that uses the errorhandlers then MPI functionality following the error. This issue is particularly difficult for applications that wish to catch informational or warning type errors (e.g., MPI_ERR_COUNT, MPI_ERR_TAG, MPI_ERR_UNSUPPORTED_OPERATION). These operations are often recoverable by the MPI implementation and/or the application. >> >> To address this portability issue, I am bringing out the MPI_ERR_CANNOT_CONTINUE error class from the stabilization proposal. I presented the idea to the MPI Forum during a plenary session last week and received a positive response on building a formal proposal [Straw vote: 22 (yes), 0 (no), 3 (abstain)]. >> >> I have created a first draft of the proposal for the working group to review on the wiki at the link below: >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue >> >> I would like to have this proposal ready by the Oct. meeting so we can have a formal plenary session on it. If all goes well, maybe we can get a first reading by Dec. >> >> Let me know what you think about this proposal. >> >> -- Josh >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://www.cs.indiana.edu/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0