September 2010 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 22 Sep '10

22 Sep '10

This proposal is not a minor change. Please do not make this hole in the standard and assume you can later add language to standardize everything that comes through the hole. If the standard is to introduce the notion of a recoverable error it must be as part of a full description of what "recovery" means. I think is is dangerous and ultimately useless to have implementors mark a failure as "recoverable" when the post error state of the distributed MPI has gone from "fully standards compliant" to "mostly standards compliant, read my user doc read my legal disclaimer, cross your fingers". See comment below for why I do not think the new hole is needed to allow people to do implementation specific recoverability. There is not even anything to prevent on implementation from deciding to add a function MPXX_WHAT_STILL_WORKS(err_code, answer) and documenting 5 or 5000 enumerated values for "answer" ranging from NOTHING through TAKE_A_CHANCE_IF_YOU_LIKE to EVERYTHING. IBM would probably return TAKE_A_CHANCE_IF_YOU_LIKE because I cannot imagine how we would promise exactly what will work and what will not but in practice most things will still work as expected. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi3-ft-bounces(a)lists.mpi-forum.org wrote on 09/21/2010 04:54:08 PM: > [image removed] > > Re: [Mpi3-ft] Defining the state of MPI after an error > > Bronis R. de Supinski > > to: > > MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > > 09/21/2010 04:59 PM > > Sent by: > > mpi3-ft-bounces(a)lists.mpi-forum.org > > Please respond to "Bronis R. de Supinski", "MPI 3.0 Fault Tolerance > and Dynamic Process Control working Group" > > > Dick: > > Re: > > The current MPI standard does not say the MPI implementation is totally > > broken once there is an error. Saying MPI state is undefined after an > > error simply says that the detailed semantic of the MPI standard can no > > longer be promised. In other words, after an error you leave behind the > > security of a portable standard semantic. You are operating at your own > > risk. You do not need to read more than that into it. > > Perhaps my problem with this position is that I come from the > background of language definitions for compilers. When you > read "undefined" in the OpenMP specification then you are > being told that things are broken and the implementation does > need to do anything or even tell you what they actually do (and > I believe the same is true for the C and C++ standards). An > alternative is "implementation defined", which requires the > implementer to document what they actually do. Without that, > you cannot even rely on actions with a specific implementation > (unless you believe "My tests so far have not failed so I am OK"). When a standard says behavior is "undefined" in some situation, it cannot mean behavior is "broken". It cannot mean the implementor is prohibited from making it still work. It cannot mean the implementor is prohibited from making certain things work and documenting them. Any statement like this in a standard would be definition of behavior and the behavior would no longer be "undefined". The only thing a standard can logically mean by "undefined" is that the STANDARD no longer mandates the definition. Bronis says: > > I strongly feel "undefined" should be reserved for situations that > mean "your program is irrevocably broken and the implementer does > not need to worry about what happens to it after encountering them." I would say this as: I strongly feel "undefined" should be reserved for situations that mean "The standard no longer guarantees your program is not irrevocably broken. The implementer is not required by the standard to worry about what happens to it after encountering them. An Implementation is free to provide any "better" behavior that may be of value but users cannot assume another implementation provides similar behavior so cannot assume standards defined portability." I do not see how the use if the word "undefined" in a standard can be interpreted as a prohibition of any behavior an implementation might offer.

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Bronis R. de Supinski 21 Sep '10

21 Sep '10

Dick: Re: > The current MPI standard does not say the MPI implementation is totally > broken once there is an error. Saying MPI state is undefined after an > error simply says that the detailed semantic of the MPI standard can no > longer be promised. In other words, after an error you leave behind the > security of a portable standard semantic. You are operating at your own > risk. You do not need to read more than that into it. Perhaps my problem with this position is that I come from the background of language definitions for compilers. When you read "undefined" in the OpenMP specification then you are being told that things are broken and the implementation does need to do anything or even tell you what they actually do (and I believe the same is true for the C and C++ standards). An alternative is "implementation defined", which requires the implementer to document what they actually do. Without that, you cannot even rely on actions with a specific implementation (unless you believe "My tests so far have not failed so I am OK"). I strongly feel "undefined" should be reserved for situations that mean "your program is irrevocably broken and the implementer does not need to worry about what happens to it after encountering them." > IBM MPI does not make any attempt to prevent future calls into libmpi > from working. If they work in the way you expect, cool. If you want to > use MPI communication in a tool that runs on its own threads and helps > with failure analysis. go for it. If you want to keep using MPI to > design an airplane wing, please warn me. I do not want to be on that > plane. MPI may look like it is still working perfectly but not be. For a > tool, it is no big deal but for an airplane wing design, it is. This position is contradictory to providing any sort of fault tolerance support with MPI. That is Josh's point and I agree. > I think having an MPI implementation return MPI_ERR_CANNOT_CONTINUE for > every MPI call after the error is way beyond what the standard intends. > The use case Bronis offered would be totally broken by this idea. But I would know that it was and I could try some alternative. Otherwise I am left to guess. > In IBM MPI, if you set MPI_ERRORS_RETURN and call MPI_Allreduce on > MPI_COMM_WORLD but have a bad MPI_Op on some, the ranks with the bad OP > will return and the ones with no error will hang forever. You will never > break the hung threads free. The threads that returned (on the ranks > with an error) can call MPI_Abort and kill the job. As I stated before that would be fine. However, you are making guarantees that are not in the standard. > On the other hand, if you already have a dup of MPI_COMM_WORLD called > MPI_tool_world and a tool thread on each task, you can communicate among > tool threads using MPI_tool_world and I would expect that to work fine. > As far as I know, everything in the MPI standard that does not depend on > MPI_COMM_WORLD will still work. An attempt to do a collective call on > MPI_COMM_WORLD may go wrong. Also, as I said before, this is even better. > Does that make passing a bad MPI_Op to MPI_Allreduce a CAN or CANNOT > CONTINUE? How would I decide? OK, now I see that your point is that Josh's proposal is too simplified and does not capture the range of possibilities. The problem is that we need to document something. Perhaps a solution is to make the state "implementation defined". A first implementation definition could be FUBAR (nothing will work). Your definition is somewhat captured above. What's more, I could go to a specific implementer and ask "Is it really hopeless or will this use case work?" I admit this is not as good as having a way to define what the situation is precisely. It is better than having no definition whatsoever. > BTW - If new functions that may be unsuccessful without damage to MPI > state are added to the standard they use a TRY semantic. Just avoid > using the return code to report that the try did not succeed. The > simplest example would be: > > MPI_TRY_ALLOC_MEM(size, info, baseptr, flag) > > If size is a negative number, this could return an error in class > MPI_ERR_ARG but it would never return an error for a memory shortage. > The flag would tell the user if the memory had been allocated. A flag of > FALSE and an RC of MPI_SUCCESS allows MPI to stay in a defined state and > the user to try again with a smaller memory request. Yes, that is a better solution. Too bad we can't apply it to existing functionality. Bronis > Dick > > > > > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Joshua Hursey <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/21/2010 12:46 PM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > ________________________________ > > > > The MPI_ERR_CANNOT_CONTINUE proposal in a small step in the direction to a more well defined set of expected/correct behavior after an error. Currently no standard MPI call can describe expected behavior after a failure since there is an overriding statement about the state being undefined. The CANNOT_CONTINUE error class defines a limit on this behavior, while still allowing the MPI implementation to decide what is correct behavior after the error for their implementation. This proposal seems small, almost inconsequential, but it opens the door a bit more for exploring well defined error semantics in future proposals. > > If an application wishes to continue using the MPI implementation after an error, it is still at their own risk. But now there is a well defined error class for the MPI implementation to hand back to the application when it does not know how to correctly proceed with this function call. The hope is that this proposal encourages implementations to start defining reasonable error semantics for specific error scenarios. These emerging error semantics are then able to be brought forward for eventual standardization. > > So for the Bcast example that you cited, I don't think I can say exactly what expected behavior should be. It is an interesting case, since even in the current standard it does not say anything about what the user can expect to happen if only one of the processes receive a local error. Does the collective fail? Does it hang? Most implementations probably just hang the other dependent processes in the collective, since doing more could hinder performance. So the user will have to have another mechanism to interrupt the collective operation (maybe a separate checking thread, or maybe relying on a tool to help detect such scenarios). This proposal just says that it is fine for the MPI library to block the erroneous process from making future calls into MPI. If the MPI implementation allows for something more reasonable to happen, it is allowed to do so. If an MPI implementation does so, and users like the semantics other implementations may be pushed to support the same semanti! > cs, and now we are on the road to standardization. > > So when someone brings forward a new proposal for an new interface (like the Shared Memory allocation routines), and they have a few error class scenarios that are informative or warning in nature they can specify what expected behavior should be for continued operation from those specific error classes. Without this proposal, defining such expected behavior in the standard is ill advised since there is the blanket statement of undefined behavior. With the proposal, it then becomes appropriate to include these semantics in the standard since the undefined behavior is able to be bound by the MPI_ERR_CANNOT_CONTINUE error class. > > So a standard compliant basic MPI implementation will always return MPI_ERR_CANNOT_CONTINUE after an error on subsequent calls. If a new interface is included in a future standard that has well defined semantics on the behavior of MPI after, say, the MPI_ERR_RANK error class. Then this basic MPI implementation will return MPI_ERR_CANNOT_CONTINUE after an error on subsequent calls except for this case in which it must behave according to the standard. > > As we are seeing with attempting to define sound semantics for the new MPI_ERR_RANK_FAIL_STOP error class in the stabilization proposal, doing so is a lot of work. So there is a challenge in attempting to define the state of MPI (and further the distributed environment) after any error class. But if it is important to applications it may be worth it. > > It is my opinion that the current standard's blanket language is too restrictive. So I see the MPI_ERR_CANNOT_CONTINUE proposal as a small step towards better error semantics. Though, as with anything, I can be convinced otherwise if there is doubt about my reasoning or the utility of such an error class. > > Thanks for your patience in this thread. > > -- Josh > > On Sep 20, 2010, at 2:34 PM, Richard Treumann wrote: > >> >> Josh >> >> Did you see this? >> >> "For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. " >> >> If my MPI implementation is going to tell you it "allows for correct operation after returning that error class", the standard needs to tell me which behavior is "correct operation" and what is not "correct operation". >> >> As I mentioned to Bronis, I have no problem with an advise to implementors urging that they at least allow an option for continued use of libmpi, after an error returns, but at the users own risk. Offhand, I cannot see any harm in urging MPI implementors to promise that all locks will be released when a subroutine returns non-SUCCESS. >> >> Can you give me a use case for the added complexity of CAN_CONTINUE vs CANNOT_CONTINUE.? >> >> Consider that an error class of CAN_CONTINUE that is returned because the MPI implementor does not really know what "correct operation" means and decides to wing it is not very useful. An MPI implementor that can get in real trouble for negligence would need to be very cautious about returning CAN_CONTINUE. >> >> Just asking the MPI implementor to refrain from flagging the error "MPI disabled after prior error" on otherwise correct future calls seems as good to me. >> >> >> Dick Treumann - MPI Team >> >> >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> >> From: Joshua Hursey <jjhursey(a)open-mpi.org> >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >> Date: 09/20/2010 01:51 PM >> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >> >> ________________________________ >> >> >> >> So the proposal makes no requirements about the state of the distributed environment after an error. All it defines is a error class to be returned by an MPI implementation once the MPI implementation can no longer function correctly. This provides a necessary feedback mechanism for the application to determine if and how the MPI implementation can be used after an error occurs. It is the responsibility of the application to avoid deadlocks and other such issues that can result from handling and recovering from errors. If an application is designed to recover from MPI_ERR_TAG that's great. If the MPI library allows for correct operation after returning that error class, then even better. If the MPI library cannot continue operation after that error then it can block subsequent operations by returning MPI_ERR_CANNOT_CONTINUE. >> >> I disagree with your assessment that this will be difficult to implement/test since a trivial implementation of this proposal is to set a global variable when an error occurs to always return MPI_ERR_CANNOT_CONTINUE when the application calls into the MPI library (it is a similar check as the 'is_mpi_initialized' check that has to be there anyway). If an implementation wants to do more (and it is definitely not required to do so) then it can define that in it's documentation. >> >> If an application wants to try to use MPI after an error it must understand that the error is local in nature (it cannot assume that every process received an error). If it can figure out how to recover from it, and the MPI implementation is able to function correctly afterward then we should let them figure it out. This allows us to define the boundaries of correct operation after an error. Before the application -could- keep using the MPI library after an error, but it was entirely undefined and not-portable what would happen. Now the application can portably attempt to use the MPI library after an error and know that it can expect either normal functionality (for those few implementations that do more than the minimum necessary) or MPI_ERR_CANNOT_CONTINUE in which the library locks them out and they then know to terminate normally. >> >> I hope this helps a bit, but maybe I am missing the core problem that you are trying to get at. >> >> -- Josh >> >> On Sep 20, 2010, at 1:09 PM, Bronis R. de Supinski wrote: >> >>> >>> Dick: >>> >>> Re: >>>> I did not intend to ignore your use case. >>> >>> No problem. >>> >>>> I did mention that I have no worries about asking MPI implementations >>>> to refrain from blocking future MPI calls after an error is detected. >>>> That was an implicit recognition of your use case. >>> >>> OK, that helps. >>> >>>> The MPI standard already forbids having an MPI call on one thread block >>>> progress on other threads. I would interpret that to include a case >>>> where a thread is blocked in a collective communication or a MPI_Recv >>>> that will never be satisfied. That is, the blocked MPI call cannot >>>> prevent other threads from using libmpi. Requiring libmpi to release >>>> any lock it took even when doing an error return would be logical but >>>> may not be implied by what is currently written. >>> >>> The current text provides no such guarantee. Once anerror is >>> returned anywhere, all bets are off (at least that is how I >>> have read it; I would need to go back through the text to >>> find the exact words that cause my concern). >>> >>>> Communicators provide a sort of isolation that keeps stray crap from >>>> failed operations from spilling over (such as eager sent message for >>>> which the MPI_Recv failed). If the tool uses its own threads and >>>> private communicators, I agree it is reasonable to ask any libmpi to >>>> avoid sabotaging that communication. >>> >>> That would be perfect from my perspective. >>> >>>> Where I get concerned is when we start talking about affirmative >>>> requirements for distributed MPI state after an error >>> >>> I don't think we can have those beyond "best effort". >>> The errors may indicate problems that make further >>> communication impossible -- perhaps because of the >>> erroneous action or just due to the state of the >>> network or other processes. I do think we can require >>> accurate return values and have an advice to implementers >>> that suggests best effort following errors. I believe >>> that would satisfy my requirements. >>> >>> Bronis >>> >>> >>> >>> >>>> >>>> Dick >>>> >>>> Dick Treumann - MPI Team >>>> IBM Systems & Technology Group >>>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>>> Tele (845) 433-7846 Fax (845) 433-8363 >>>> >>>> >>>> >>>> From: "Bronis R. de Supinski" <bronis(a)llnl.gov> >>>> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >>>> Date: 09/20/2010 12:46 PM >>>> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >>>> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >>>> >>>> ________________________________ >>>> >>>> >>>> >>>> >>>> Dick: >>>> >>>> You seem to be ignoring my use case. Specifically, I >>>> have tool threads that use MPI. Their use of MPI should >>>> be unaffected by all of the scenarios that you are raising. >>>> However, the standard provides no way for me to tell if >>>> they work correctly in these situations. I just have to >>>> cross my fingers and hope. >>>> >>>> FYI: Your implementation has long met this requirement >>>> (my hopes are not dashed with it). Others have begun to >>>> recently. In any event, I would like some way to tell... >>>> >>>> Further, it is useful in many other scenarios apply to know >>>> that the implementation intends to remain usable. I am not >>>> looking for a promise of correct execution; I am looking >>>> for a promise of best effort and accurate return codes. >>>> >>>> Bronis >>>> >>>> >>>> >>>> On Mon, 20 Sep 2010, Richard Treumann wrote: >>>> >>>>> >>>>> If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) >>>>> >>>>> MPI_Abort, >>>>> MPI_Error_string >>>>> MPI_Error_class >>>>> >>>>> I assume it should be corrected as a trivial oversight in the original text. >>>>> >>>>> I would regard the real issue as being the difficulty with assuring the state of remote processes. >>>>> >>>>> There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. >>>>> >>>>> For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Dick Treumann - MPI Team >>>>> IBM Systems & Technology Group >>>>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>>>> Tele (845) 433-7846 Fax (845) 433-8363 >>>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft<http://BLOCKEDblockedlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft><http://BLOCKEDblockedlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >>>> >>>> >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://BLOCKEDwww.BLOCKEDcs.indiana.edu/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> <ATT00001..txt> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://BLOCKEDwww.BLOCKEDcs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 21 Sep '10

21 Sep '10

The current MPI standard does not say the MPI implementation is totally broken once there is an error. Saying MPI state is undefined after an error simply says that the detailed semantic of the MPI standard can no longer be promised. In other words, after an error you leave behind the security of a portable standard semantic. You are operating at your own risk. You do not need to read more than that into it. IBM MPI does not make any attempt to prevent future calls into libmpi from working. If they work in the way you expect, cool. If you want to use MPI communication in a tool that runs on its own threads and helps with failure analysis. go for it. If you want to keep using MPI to design an airplane wing, please warn me. I do not want to be on that plane. MPI may look like it is still working perfectly but not be. For a tool, it is no big deal but for an airplane wing design, it is. I think having an MPI implementation return MPI_ERR_CANNOT_CONTINUE for every MPI call after the error is way beyond what the standard intends. The use case Bronis offered would be totally broken by this idea. In IBM MPI, if you set MPI_ERRORS_RETURN and call MPI_Allreduce on MPI_COMM_WORLD but have a bad MPI_Op on some, the ranks with the bad OP will return and the ones with no error will hang forever. You will never break the hung threads free. The threads that returned (on the ranks with an error) can call MPI_Abort and kill the job. On the other hand, if you already have a dup of MPI_COMM_WORLD called MPI_tool_world and a tool thread on each task, you can communicate among tool threads using MPI_tool_world and I would expect that to work fine. As far as I know, everything in the MPI standard that does not depend on MPI_COMM_WORLD will still work. An attempt to do a collective call on MPI_COMM_WORLD may go wrong. Does that make passing a bad MPI_Op to MPI_Allreduce a CAN or CANNOT CONTINUE? How would I decide? BTW - If new functions that may be unsuccessful without damage to MPI state are added to the standard they use a TRY semantic. Just avoid using the return code to report that the try did not succeed. The simplest example would be: MPI_TRY_ALLOC_MEM(size, info, baseptr, flag) If size is a negative number, this could return an error in class MPI_ERR_ARG but it would never return an error for a memory shortage. The flag would tell the user if the memory had been allocated. A flag of FALSE and an RC of MPI_SUCCESS allows MPI to stay in a defined state and the user to try again with a smaller memory request. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Joshua Hursey <jjhursey(a)open-mpi.org> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/21/2010 12:46 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org The MPI_ERR_CANNOT_CONTINUE proposal in a small step in the direction to a more well defined set of expected/correct behavior after an error. Currently no standard MPI call can describe expected behavior after a failure since there is an overriding statement about the state being undefined. The CANNOT_CONTINUE error class defines a limit on this behavior, while still allowing the MPI implementation to decide what is correct behavior after the error for their implementation. This proposal seems small, almost inconsequential, but it opens the door a bit more for exploring well defined error semantics in future proposals. If an application wishes to continue using the MPI implementation after an error, it is still at their own risk. But now there is a well defined error class for the MPI implementation to hand back to the application when it does not know how to correctly proceed with this function call. The hope is that this proposal encourages implementations to start defining reasonable error semantics for specific error scenarios. These emerging error semantics are then able to be brought forward for eventual standardization. So for the Bcast example that you cited, I don't think I can say exactly what expected behavior should be. It is an interesting case, since even in the current standard it does not say anything about what the user can expect to happen if only one of the processes receive a local error. Does the collective fail? Does it hang? Most implementations probably just hang the other dependent processes in the collective, since doing more could hinder performance. So the user will have to have another mechanism to interrupt the collective operation (maybe a separate checking thread, or maybe relying on a tool to help detect such scenarios). This proposal just says that it is fine for the MPI library to block the erroneous process from making future calls into MPI. If the MPI implementation allows for something more reasonable to happen, it is allowed to do so. If an MPI implementation does so, and users like the semantics other implementations may be pushed to support the same semanti! cs, and now we are on the road to standardization. So when someone brings forward a new proposal for an new interface (like the Shared Memory allocation routines), and they have a few error class scenarios that are informative or warning in nature they can specify what expected behavior should be for continued operation from those specific error classes. Without this proposal, defining such expected behavior in the standard is ill advised since there is the blanket statement of undefined behavior. With the proposal, it then becomes appropriate to include these semantics in the standard since the undefined behavior is able to be bound by the MPI_ERR_CANNOT_CONTINUE error class. So a standard compliant basic MPI implementation will always return MPI_ERR_CANNOT_CONTINUE after an error on subsequent calls. If a new interface is included in a future standard that has well defined semantics on the behavior of MPI after, say, the MPI_ERR_RANK error class. Then this basic MPI implementation will return MPI_ERR_CANNOT_CONTINUE after an error on subsequent calls except for this case in which it must behave according to the standard. As we are seeing with attempting to define sound semantics for the new MPI_ERR_RANK_FAIL_STOP error class in the stabilization proposal, doing so is a lot of work. So there is a challenge in attempting to define the state of MPI (and further the distributed environment) after any error class. But if it is important to applications it may be worth it. It is my opinion that the current standard's blanket language is too restrictive. So I see the MPI_ERR_CANNOT_CONTINUE proposal as a small step towards better error semantics. Though, as with anything, I can be convinced otherwise if there is doubt about my reasoning or the utility of such an error class. Thanks for your patience in this thread. -- Josh On Sep 20, 2010, at 2:34 PM, Richard Treumann wrote: > > Josh > > Did you see this? > > "For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. " > > If my MPI implementation is going to tell you it "allows for correct operation after returning that error class", the standard needs to tell me which behavior is "correct operation" and what is not "correct operation". > > As I mentioned to Bronis, I have no problem with an advise to implementors urging that they at least allow an option for continued use of libmpi, after an error returns, but at the users own risk. Offhand, I cannot see any harm in urging MPI implementors to promise that all locks will be released when a subroutine returns non-SUCCESS. > > Can you give me a use case for the added complexity of CAN_CONTINUE vs CANNOT_CONTINUE.? > > Consider that an error class of CAN_CONTINUE that is returned because the MPI implementor does not really know what "correct operation" means and decides to wing it is not very useful. An MPI implementor that can get in real trouble for negligence would need to be very cautious about returning CAN_CONTINUE. > > Just asking the MPI implementor to refrain from flagging the error "MPI disabled after prior error" on otherwise correct future calls seems as good to me. > > > Dick Treumann - MPI Team > > > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Joshua Hursey <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/20/2010 01:51 PM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > ________________________________ > > > > So the proposal makes no requirements about the state of the distributed environment after an error. All it defines is a error class to be returned by an MPI implementation once the MPI implementation can no longer function correctly. This provides a necessary feedback mechanism for the application to determine if and how the MPI implementation can be used after an error occurs. It is the responsibility of the application to avoid deadlocks and other such issues that can result from handling and recovering from errors. If an application is designed to recover from MPI_ERR_TAG that's great. If the MPI library allows for correct operation after returning that error class, then even better. If the MPI library cannot continue operation after that error then it can block subsequent operations by returning MPI_ERR_CANNOT_CONTINUE. > > I disagree with your assessment that this will be difficult to implement/test since a trivial implementation of this proposal is to set a global variable when an error occurs to always return MPI_ERR_CANNOT_CONTINUE when the application calls into the MPI library (it is a similar check as the 'is_mpi_initialized' check that has to be there anyway). If an implementation wants to do more (and it is definitely not required to do so) then it can define that in it's documentation. > > If an application wants to try to use MPI after an error it must understand that the error is local in nature (it cannot assume that every process received an error). If it can figure out how to recover from it, and the MPI implementation is able to function correctly afterward then we should let them figure it out. This allows us to define the boundaries of correct operation after an error. Before the application -could- keep using the MPI library after an error, but it was entirely undefined and not-portable what would happen. Now the application can portably attempt to use the MPI library after an error and know that it can expect either normal functionality (for those few implementations that do more than the minimum necessary) or MPI_ERR_CANNOT_CONTINUE in which the library locks them out and they then know to terminate normally. > > I hope this helps a bit, but maybe I am missing the core problem that you are trying to get at. > > -- Josh > > On Sep 20, 2010, at 1:09 PM, Bronis R. de Supinski wrote: > >> >> Dick: >> >> Re: >>> I did not intend to ignore your use case. >> >> No problem. >> >>> I did mention that I have no worries about asking MPI implementations >>> to refrain from blocking future MPI calls after an error is detected. >>> That was an implicit recognition of your use case. >> >> OK, that helps. >> >>> The MPI standard already forbids having an MPI call on one thread block >>> progress on other threads. I would interpret that to include a case >>> where a thread is blocked in a collective communication or a MPI_Recv >>> that will never be satisfied. That is, the blocked MPI call cannot >>> prevent other threads from using libmpi. Requiring libmpi to release >>> any lock it took even when doing an error return would be logical but >>> may not be implied by what is currently written. >> >> The current text provides no such guarantee. Once anerror is >> returned anywhere, all bets are off (at least that is how I >> have read it; I would need to go back through the text to >> find the exact words that cause my concern). >> >>> Communicators provide a sort of isolation that keeps stray crap from >>> failed operations from spilling over (such as eager sent message for >>> which the MPI_Recv failed). If the tool uses its own threads and >>> private communicators, I agree it is reasonable to ask any libmpi to >>> avoid sabotaging that communication. >> >> That would be perfect from my perspective. >> >>> Where I get concerned is when we start talking about affirmative >>> requirements for distributed MPI state after an error >> >> I don't think we can have those beyond "best effort". >> The errors may indicate problems that make further >> communication impossible -- perhaps because of the >> erroneous action or just due to the state of the >> network or other processes. I do think we can require >> accurate return values and have an advice to implementers >> that suggests best effort following errors. I believe >> that would satisfy my requirements. >> >> Bronis >> >> >> >> >>> >>> Dick >>> >>> Dick Treumann - MPI Team >>> IBM Systems & Technology Group >>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>> Tele (845) 433-7846 Fax (845) 433-8363 >>> >>> >>> >>> From: "Bronis R. de Supinski" <bronis(a)llnl.gov> >>> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >>> Date: 09/20/2010 12:46 PM >>> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >>> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >>> >>> ________________________________ >>> >>> >>> >>> >>> Dick: >>> >>> You seem to be ignoring my use case. Specifically, I >>> have tool threads that use MPI. Their use of MPI should >>> be unaffected by all of the scenarios that you are raising. >>> However, the standard provides no way for me to tell if >>> they work correctly in these situations. I just have to >>> cross my fingers and hope. >>> >>> FYI: Your implementation has long met this requirement >>> (my hopes are not dashed with it). Others have begun to >>> recently. In any event, I would like some way to tell... >>> >>> Further, it is useful in many other scenarios apply to know >>> that the implementation intends to remain usable. I am not >>> looking for a promise of correct execution; I am looking >>> for a promise of best effort and accurate return codes. >>> >>> Bronis >>> >>> >>> >>> On Mon, 20 Sep 2010, Richard Treumann wrote: >>> >>>> >>>> If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) >>>> >>>> MPI_Abort, >>>> MPI_Error_string >>>> MPI_Error_class >>>> >>>> I assume it should be corrected as a trivial oversight in the original text. >>>> >>>> I would regard the real issue as being the difficulty with assuring the state of remote processes. >>>> >>>> There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. >>>> >>>> For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. >>>> >>>> >>>> >>>> >>>> >>>> >>>> Dick Treumann - MPI Team >>>> IBM Systems & Technology Group >>>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>>> Tele (845) 433-7846 Fax (845) 433-8363 >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft< http://blockedlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >>> >>> >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > <ATT00001..txt> ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Joshua Hursey 21 Sep '10

21 Sep '10

The MPI_ERR_CANNOT_CONTINUE proposal in a small step in the direction to a more well defined set of expected/correct behavior after an error. Currently no standard MPI call can describe expected behavior after a failure since there is an overriding statement about the state being undefined. The CANNOT_CONTINUE error class defines a limit on this behavior, while still allowing the MPI implementation to decide what is correct behavior after the error for their implementation. This proposal seems small, almost inconsequential, but it opens the door a bit more for exploring well defined error semantics in future proposals. If an application wishes to continue using the MPI implementation after an error, it is still at their own risk. But now there is a well defined error class for the MPI implementation to hand back to the application when it does not know how to correctly proceed with this function call. The hope is that this proposal encourages implementations to start defining reasonable error semantics for specific error scenarios. These emerging error semantics are then able to be brought forward for eventual standardization. So for the Bcast example that you cited, I don't think I can say exactly what expected behavior should be. It is an interesting case, since even in the current standard it does not say anything about what the user can expect to happen if only one of the processes receive a local error. Does the collective fail? Does it hang? Most implementations probably just hang the other dependent processes in the collective, since doing more could hinder performance. So the user will have to have another mechanism to interrupt the collective operation (maybe a separate checking thread, or maybe relying on a tool to help detect such scenarios). This proposal just says that it is fine for the MPI library to block the erroneous process from making future calls into MPI. If the MPI implementation allows for something more reasonable to happen, it is allowed to do so. If an MPI implementation does so, and users like the semantics other implementations may be pushed to support the same semantics, and now we are on the road to standardization. So when someone brings forward a new proposal for an new interface (like the Shared Memory allocation routines), and they have a few error class scenarios that are informative or warning in nature they can specify what expected behavior should be for continued operation from those specific error classes. Without this proposal, defining such expected behavior in the standard is ill advised since there is the blanket statement of undefined behavior. With the proposal, it then becomes appropriate to include these semantics in the standard since the undefined behavior is able to be bound by the MPI_ERR_CANNOT_CONTINUE error class. So a standard compliant basic MPI implementation will always return MPI_ERR_CANNOT_CONTINUE after an error on subsequent calls. If a new interface is included in a future standard that has well defined semantics on the behavior of MPI after, say, the MPI_ERR_RANK error class. Then this basic MPI implementation will return MPI_ERR_CANNOT_CONTINUE after an error on subsequent calls except for this case in which it must behave according to the standard. As we are seeing with attempting to define sound semantics for the new MPI_ERR_RANK_FAIL_STOP error class in the stabilization proposal, doing so is a lot of work. So there is a challenge in attempting to define the state of MPI (and further the distributed environment) after any error class. But if it is important to applications it may be worth it. It is my opinion that the current standard's blanket language is too restrictive. So I see the MPI_ERR_CANNOT_CONTINUE proposal as a small step towards better error semantics. Though, as with anything, I can be convinced otherwise if there is doubt about my reasoning or the utility of such an error class. Thanks for your patience in this thread. -- Josh On Sep 20, 2010, at 2:34 PM, Richard Treumann wrote: > > Josh > > Did you see this? > > "For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. " > > If my MPI implementation is going to tell you it "allows for correct operation after returning that error class", the standard needs to tell me which behavior is "correct operation" and what is not "correct operation". > > As I mentioned to Bronis, I have no problem with an advise to implementors urging that they at least allow an option for continued use of libmpi, after an error returns, but at the users own risk. Offhand, I cannot see any harm in urging MPI implementors to promise that all locks will be released when a subroutine returns non-SUCCESS. > > Can you give me a use case for the added complexity of CAN_CONTINUE vs CANNOT_CONTINUE.? > > Consider that an error class of CAN_CONTINUE that is returned because the MPI implementor does not really know what "correct operation" means and decides to wing it is not very useful. An MPI implementor that can get in real trouble for negligence would need to be very cautious about returning CAN_CONTINUE. > > Just asking the MPI implementor to refrain from flagging the error "MPI disabled after prior error" on otherwise correct future calls seems as good to me. > > > Dick Treumann - MPI Team > > > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > > From: Joshua Hursey <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: 09/20/2010 01:51 PM > Subject: Re: [Mpi3-ft] Defining the state of MPI after an error > Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org > > ________________________________ > > > > So the proposal makes no requirements about the state of the distributed environment after an error. All it defines is a error class to be returned by an MPI implementation once the MPI implementation can no longer function correctly. This provides a necessary feedback mechanism for the application to determine if and how the MPI implementation can be used after an error occurs. It is the responsibility of the application to avoid deadlocks and other such issues that can result from handling and recovering from errors. If an application is designed to recover from MPI_ERR_TAG that's great. If the MPI library allows for correct operation after returning that error class, then even better. If the MPI library cannot continue operation after that error then it can block subsequent operations by returning MPI_ERR_CANNOT_CONTINUE. > > I disagree with your assessment that this will be difficult to implement/test since a trivial implementation of this proposal is to set a global variable when an error occurs to always return MPI_ERR_CANNOT_CONTINUE when the application calls into the MPI library (it is a similar check as the 'is_mpi_initialized' check that has to be there anyway). If an implementation wants to do more (and it is definitely not required to do so) then it can define that in it's documentation. > > If an application wants to try to use MPI after an error it must understand that the error is local in nature (it cannot assume that every process received an error). If it can figure out how to recover from it, and the MPI implementation is able to function correctly afterward then we should let them figure it out. This allows us to define the boundaries of correct operation after an error. Before the application -could- keep using the MPI library after an error, but it was entirely undefined and not-portable what would happen. Now the application can portably attempt to use the MPI library after an error and know that it can expect either normal functionality (for those few implementations that do more than the minimum necessary) or MPI_ERR_CANNOT_CONTINUE in which the library locks them out and they then know to terminate normally. > > I hope this helps a bit, but maybe I am missing the core problem that you are trying to get at. > > -- Josh > > On Sep 20, 2010, at 1:09 PM, Bronis R. de Supinski wrote: > >> >> Dick: >> >> Re: >>> I did not intend to ignore your use case. >> >> No problem. >> >>> I did mention that I have no worries about asking MPI implementations >>> to refrain from blocking future MPI calls after an error is detected. >>> That was an implicit recognition of your use case. >> >> OK, that helps. >> >>> The MPI standard already forbids having an MPI call on one thread block >>> progress on other threads. I would interpret that to include a case >>> where a thread is blocked in a collective communication or a MPI_Recv >>> that will never be satisfied. That is, the blocked MPI call cannot >>> prevent other threads from using libmpi. Requiring libmpi to release >>> any lock it took even when doing an error return would be logical but >>> may not be implied by what is currently written. >> >> The current text provides no such guarantee. Once anerror is >> returned anywhere, all bets are off (at least that is how I >> have read it; I would need to go back through the text to >> find the exact words that cause my concern). >> >>> Communicators provide a sort of isolation that keeps stray crap from >>> failed operations from spilling over (such as eager sent message for >>> which the MPI_Recv failed). If the tool uses its own threads and >>> private communicators, I agree it is reasonable to ask any libmpi to >>> avoid sabotaging that communication. >> >> That would be perfect from my perspective. >> >>> Where I get concerned is when we start talking about affirmative >>> requirements for distributed MPI state after an error >> >> I don't think we can have those beyond "best effort". >> The errors may indicate problems that make further >> communication impossible -- perhaps because of the >> erroneous action or just due to the state of the >> network or other processes. I do think we can require >> accurate return values and have an advice to implementers >> that suggests best effort following errors. I believe >> that would satisfy my requirements. >> >> Bronis >> >> >> >> >>> >>> Dick >>> >>> Dick Treumann - MPI Team >>> IBM Systems & Technology Group >>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>> Tele (845) 433-7846 Fax (845) 433-8363 >>> >>> >>> >>> From: "Bronis R. de Supinski" <bronis(a)llnl.gov> >>> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >>> Date: 09/20/2010 12:46 PM >>> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >>> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >>> >>> ________________________________ >>> >>> >>> >>> >>> Dick: >>> >>> You seem to be ignoring my use case. Specifically, I >>> have tool threads that use MPI. Their use of MPI should >>> be unaffected by all of the scenarios that you are raising. >>> However, the standard provides no way for me to tell if >>> they work correctly in these situations. I just have to >>> cross my fingers and hope. >>> >>> FYI: Your implementation has long met this requirement >>> (my hopes are not dashed with it). Others have begun to >>> recently. In any event, I would like some way to tell... >>> >>> Further, it is useful in many other scenarios apply to know >>> that the implementation intends to remain usable. I am not >>> looking for a promise of correct execution; I am looking >>> for a promise of best effort and accurate return codes. >>> >>> Bronis >>> >>> >>> >>> On Mon, 20 Sep 2010, Richard Treumann wrote: >>> >>>> >>>> If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) >>>> >>>> MPI_Abort, >>>> MPI_Error_string >>>> MPI_Error_class >>>> >>>> I assume it should be corrected as a trivial oversight in the original text. >>>> >>>> I would regard the real issue as being the difficulty with assuring the state of remote processes. >>>> >>>> There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. >>>> >>>> For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. >>>> >>>> >>>> >>>> >>>> >>>> >>>> Dick Treumann - MPI Team >>>> IBM Systems & Technology Group >>>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>>> Tele (845) 433-7846 Fax (845) 433-8363 >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft<http://blockedlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >>> >>> >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > <ATT00001..txt> ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question
by Graham, Richard L. 21 Sep '10

21 Sep '10

Not being that specific. What I am suggesting is that comm_size return the number actually started. If we let MPI try to start M ranks and consider it OK if only N were started where N < M, than perhaps we should expose other relevant information, perhaps the value of M. Rich On 9/21/10 11:55 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: In the last sentence, you say you're "suggesting we standardize what these do." What is "these" referring to? Are you suggesting we add to the standard the _concept_ of hard and soft process counts for the runtime (as opposed to the parameters themselves)? -d On Sep 20, 2010, at 7:50 PM, Graham, Richard L. wrote: > I mean that if we don't clearly define what the mpi_comm_size() returns and have it be standardized, it then becomes something that users can't depend on, since it is implementation specific. > > Not suggesting standardizing on the command line parameters - we don't now, and still all understand the same thing that comm_size returns. I am suggesting we standardize what these do, and if we need more data than we have now, add the capability to ask for that information. > > Rich > > > On 9/20/10 6:59 PM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: > > > > Sorry, I didn't understand your last sentence. What did you mean by "pick your meaning of a return value"? I might have missed that in a previous email. > > Are you advocating standardizing the command line parameters (-n and -soft)? Or a way to query them? > > -d > > On Sep 20, 2010, at 1:28 PM, Graham, Richard L. wrote: > >> My objection is not defining what the value of mpi_comm_size() returns. I think that it should return a well define value, and I would advocate for the number of ranks that have been started, which is what it returns today. If we want to be able to have users ask about command line options - soft limits, hard limits, or ?, we should provide a standard way to ask this question. I may have missed what you are proposing, but "pick your meaning of a return value" seems like it would lead to non-portable apps. >> >> Rich >> >> >> On 9/20/10 10:51 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: >> >> How would it be specified? >> >> "To start an MPI application the user should call "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or will fail with error code XXX." >> >> This would make any runtime that provides the -soft parameter noncompliant. Do we really want to do this? >> >> Alternatively, we could define it something like: >> >> "To start an MPI application the user should cal "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or another number or MPI processes depending on other parameters specified by the user, or fail with an error code." >> >> But isn't this what we have now? What's wrong with that? >> >> -d >> >> On Sep 20, 2010, at 9:39 AM, Graham, Richard L. wrote: >> >>> I strongly disagree here. This is such a basic bit of data, we clearly need to specify the output. >>> >>> Rich >>> >>> ----- Original Message ----- >>> From: Joshua Hursey [mailto:[email protected]] >>> Sent: Monday, September 20, 2010 10:09 AM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group <mpi3-ft(a)lists.mpi-forum.org> >>> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >>> >>> This is an interesting discussion (I am glad it is archived). >>> >>> I feel that Darius makes a strong argument about leaving it undefined in the standard for many of the reasons that have been mentioned. I think an implementation can decide then if one way is more important to their users than another. It is my opinion that defining it does not gain us much since most applications base their initialization on the result of MPI_Comm_size() instead of requiring a specific number of processes. Further, if they do require a specific number, they can interpret the results of MPI_Comm_size() and choose to abort or work around the difference. >>> >>> -- Josh >>> >>> On Sep 17, 2010, at 11:00 AM, Graham, Richard L. wrote: >>> >>>> Why would we select one behaviour over another ? Right now users use the value returned from mpi_comm_size() to determine how to setup their parallel job, so telling them what the actually got seems like the most obvious return value to have, especially if we do not provide a way to get both what they asked for, and what they actually got. Having this be implementation specific does not seem to be prudent, since they would have to know what implementation they are using to figure out what to do with the data. I believe this is very different than an implementation choice to use an eager protocol for large message vs. a rendezvous which does not have correctness implications. >>>> >>>> Also, at startup the user has not set anything up yet, with respect to MPI, so forming a dense communicator is well defined and the easiest thing to do. Once failure occurs, I will argue exactly the opposite, based on the outcome of such a decisions, and the fact that we will provide them a way within MPI to figure out what has happened. >>>> >>>> Rich >>>> >>>> On 9/17/10 10:14 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: >>>> >>>> >>>> >>>> I don't think we need to choose one or the other (in fact I feel strongly that we should not force one behavior or the other). The choice to have MPI_COMM_WORLD contain N or M processes (or failing if it can't get all N) is implementation dependent. Presumably the behavior would be user selectable (e.g., using a -soft option or something similar to mpiexec). >>>> >>>> The user would use the mechanisms we will provide to deal with any dead processes (e.g., validating a communicator, etc). >>>> >>>> -d >>>> >>>> On Sep 17, 2010, at 4:02 PM, Bronevetsky, Greg wrote: >>>> >>>>> I agree that MPI_Comm_size should return the number of ranks in the communicator regardless of whether they're operational or not. However, this just pushes the question further back: if the user asked for N processes but only M have started up, how many ranks should MPI_COMM_WORLD have? Either choice is going to be self-consistent from MPI's point of view. If its N, then some ranks will be dead. If it is M then the application may not have enough processes to work with. The former (N) case has property that the user doesn't need to add code to check for this condition since their existing error checking code will catch this situation. The latter case (M) is nice because it cheaper to check whether we got fewer processes than expected than to explicitly try to communicate with them. >>>>> >>>>> As such, I don't see a strong motivation for choosing either. However, we should just pick one and stick with it to avoid unnecessary API divergence. >>>>> >>>>> Greg Bronevetsky >>>>> Lawrence Livermore National Lab >>>>> (925) 424-5756 >>>>> bronevetsky(a)llnl.gov >>>>> http://greg.bronevetsky.com >>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Bronis R. de Supinski >>>>>> Sent: Friday, September 17, 2010 5:23 AM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >>>>>> >>>>>> >>>>>> I agree with Rich and Darius. >>>>>> >>>>>> On Fri, 17 Sep 2010, Darius Buntinas wrote: >>>>>> >>>>>>> >>>>>>> I don't think we should change the standard in this case. For >>>>>> MPI_Comm_size to have any useful meaning, it needs to return the size of >>>>>> the communicator: i.e., if comm_size returns N, you should be able to do a >>>>>> send to processes 0 through N-1. Of course if some of those processes have >>>>>> failed, you'll get an error associated with the process failure, but never >>>>>> an error for an invalid rank. >>>>>>> >>>>>>> As discussed in the section about mpiexec, an implementation may decide >>>>>> to provide a soft process count argument. So "mpiexec -n 10 -soft 5 ./cpi" >>>>>> can start any number between 5 and 10 processes. But that does not affect >>>>>> the meaning of the size of MPI_COMM_WORLD: regardless of the number of >>>>>> processes the implementation decides to start, MPI_Comm_size will return >>>>>> the _actual_ number of processes started. >>>>>>> >>>>>>> -d >>>>>>> >>>>>>> On Sep 17, 2010, at 11:22 AM, Graham, Richard L. wrote: >>>>>>> >>>>>>>> We need to clearly define what N or M is and not leave it to the >>>>>> implementation. 100% of the codes that seen over the past 15 years that >>>>>> check this value use it to indicate how many process have started. Any >>>>>> thing else is really useless, aside from letting the user find out how many >>>>>> processes actually started up, and then know how many did not start up. >>>>>>>> >>>>>>>> Rich >>>>>>>> >>>>>>>> >>>>>>>> On 9/17/10 4:27 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: >>>>>>>> >>>>>>>> So the Run-Through Stabilization proposal has been updated per our >>>>>> discussion in the working group meeting at the MPI Forum. The changes are >>>>>> summarized below: >>>>>>>> - Add a Legacy Library Support example >>>>>>>> - Clarify new error classes >>>>>>>> - Update the MPI_Init and MPI_Finalize wording to be simpler and more >>>>>> direct. >>>>>>>> - Fix wording of group creation calls versus communicator creation >>>>>> calls. >>>>>>>> >>>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>>> web/wiki/ft/run_through_stabilization >>>>>>>> >>>>>>>> >>>>>>>> One question that we discussed quite a bit during the meeting was the >>>>>> issue of the return value of MPI_Comm_size() when processes fail during >>>>>> launch. I attempted to capture the discussion in the room in the Open >>>>>> Question attached to the discussion of MPI_Init: >>>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>>> web/wiki/ft/run_through_stabilization#MPI_INIT >>>>>>>> >>>>>>>> Open question: >>>>>>>> If the user asks to start N processes on the command line, and only M >>>>>> processes were successfully launched (where M < N), then what should be >>>>>> returned from MPI_COMM_SIZE? >>>>>>>> >>>>>>>> The return value must be consistent across all alive members of the >>>>>> group. The issue is if it should return N or M. >>>>>>>> >>>>>>>> The feeling in the room was that since the MPI standard does not define >>>>>> the ability for the user to ask for a specific number of processes before >>>>>> initthen it is hard to define that this is the number it should be. >>>>>>>> >>>>>>>> So it is left to the implementation whether it is M or N. If it is M, >>>>>> then the user has other techniques to find out what it originally asked for >>>>>> (e.g., by passing it as a command line argument to the application itself). >>>>>>>> >>>>>>>> >>>>>>>> What do people think about the MPI_Comm_size issue? >>>>>>>> >>>>>>>> -- Josh >>>>>>>> >>>>>>>> ------------------------------------ >>>>>>>> Joshua Hursey >>>>>>>> Postdoctoral Research Associate >>>>>>>> Oak Ridge National Laboratory >>>>>>>> http://BLOCKEDBLOCKEDwww.BLOCKEDBLOCKEDcs.indiana.edu/~jjhursey >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> >>> ------------------------------------ >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://www.cs.indiana.edu/~jjhursey >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question
by Darius Buntinas 21 Sep '10

21 Sep '10

In the last sentence, you say you're "suggesting we standardize what these do." What is "these" referring to? Are you suggesting we add to the standard the _concept_ of hard and soft process counts for the runtime (as opposed to the parameters themselves)? -d On Sep 20, 2010, at 7:50 PM, Graham, Richard L. wrote: > I mean that if we don't clearly define what the mpi_comm_size() returns and have it be standardized, it then becomes something that users can't depend on, since it is implementation specific. > > Not suggesting standardizing on the command line parameters - we don't now, and still all understand the same thing that comm_size returns. I am suggesting we standardize what these do, and if we need more data than we have now, add the capability to ask for that information. > > Rich > > > On 9/20/10 6:59 PM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: > > > > Sorry, I didn't understand your last sentence. What did you mean by "pick your meaning of a return value"? I might have missed that in a previous email. > > Are you advocating standardizing the command line parameters (-n and -soft)? Or a way to query them? > > -d > > On Sep 20, 2010, at 1:28 PM, Graham, Richard L. wrote: > >> My objection is not defining what the value of mpi_comm_size() returns. I think that it should return a well define value, and I would advocate for the number of ranks that have been started, which is what it returns today. If we want to be able to have users ask about command line options - soft limits, hard limits, or ?, we should provide a standard way to ask this question. I may have missed what you are proposing, but "pick your meaning of a return value" seems like it would lead to non-portable apps. >> >> Rich >> >> >> On 9/20/10 10:51 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: >> >> How would it be specified? >> >> "To start an MPI application the user should call "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or will fail with error code XXX." >> >> This would make any runtime that provides the -soft parameter noncompliant. Do we really want to do this? >> >> Alternatively, we could define it something like: >> >> "To start an MPI application the user should cal "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or another number or MPI processes depending on other parameters specified by the user, or fail with an error code." >> >> But isn't this what we have now? What's wrong with that? >> >> -d >> >> On Sep 20, 2010, at 9:39 AM, Graham, Richard L. wrote: >> >>> I strongly disagree here. This is such a basic bit of data, we clearly need to specify the output. >>> >>> Rich >>> >>> ----- Original Message ----- >>> From: Joshua Hursey [mailto:[email protected]] >>> Sent: Monday, September 20, 2010 10:09 AM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group <mpi3-ft(a)lists.mpi-forum.org> >>> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >>> >>> This is an interesting discussion (I am glad it is archived). >>> >>> I feel that Darius makes a strong argument about leaving it undefined in the standard for many of the reasons that have been mentioned. I think an implementation can decide then if one way is more important to their users than another. It is my opinion that defining it does not gain us much since most applications base their initialization on the result of MPI_Comm_size() instead of requiring a specific number of processes. Further, if they do require a specific number, they can interpret the results of MPI_Comm_size() and choose to abort or work around the difference. >>> >>> -- Josh >>> >>> On Sep 17, 2010, at 11:00 AM, Graham, Richard L. wrote: >>> >>>> Why would we select one behaviour over another ? Right now users use the value returned from mpi_comm_size() to determine how to setup their parallel job, so telling them what the actually got seems like the most obvious return value to have, especially if we do not provide a way to get both what they asked for, and what they actually got. Having this be implementation specific does not seem to be prudent, since they would have to know what implementation they are using to figure out what to do with the data. I believe this is very different than an implementation choice to use an eager protocol for large message vs. a rendezvous which does not have correctness implications. >>>> >>>> Also, at startup the user has not set anything up yet, with respect to MPI, so forming a dense communicator is well defined and the easiest thing to do. Once failure occurs, I will argue exactly the opposite, based on the outcome of such a decisions, and the fact that we will provide them a way within MPI to figure out what has happened. >>>> >>>> Rich >>>> >>>> On 9/17/10 10:14 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: >>>> >>>> >>>> >>>> I don't think we need to choose one or the other (in fact I feel strongly that we should not force one behavior or the other). The choice to have MPI_COMM_WORLD contain N or M processes (or failing if it can't get all N) is implementation dependent. Presumably the behavior would be user selectable (e.g., using a -soft option or something similar to mpiexec). >>>> >>>> The user would use the mechanisms we will provide to deal with any dead processes (e.g., validating a communicator, etc). >>>> >>>> -d >>>> >>>> On Sep 17, 2010, at 4:02 PM, Bronevetsky, Greg wrote: >>>> >>>>> I agree that MPI_Comm_size should return the number of ranks in the communicator regardless of whether they're operational or not. However, this just pushes the question further back: if the user asked for N processes but only M have started up, how many ranks should MPI_COMM_WORLD have? Either choice is going to be self-consistent from MPI's point of view. If its N, then some ranks will be dead. If it is M then the application may not have enough processes to work with. The former (N) case has property that the user doesn't need to add code to check for this condition since their existing error checking code will catch this situation. The latter case (M) is nice because it cheaper to check whether we got fewer processes than expected than to explicitly try to communicate with them. >>>>> >>>>> As such, I don't see a strong motivation for choosing either. However, we should just pick one and stick with it to avoid unnecessary API divergence. >>>>> >>>>> Greg Bronevetsky >>>>> Lawrence Livermore National Lab >>>>> (925) 424-5756 >>>>> bronevetsky(a)llnl.gov >>>>> http://greg.bronevetsky.com >>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Bronis R. de Supinski >>>>>> Sent: Friday, September 17, 2010 5:23 AM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >>>>>> >>>>>> >>>>>> I agree with Rich and Darius. >>>>>> >>>>>> On Fri, 17 Sep 2010, Darius Buntinas wrote: >>>>>> >>>>>>> >>>>>>> I don't think we should change the standard in this case. For >>>>>> MPI_Comm_size to have any useful meaning, it needs to return the size of >>>>>> the communicator: i.e., if comm_size returns N, you should be able to do a >>>>>> send to processes 0 through N-1. Of course if some of those processes have >>>>>> failed, you'll get an error associated with the process failure, but never >>>>>> an error for an invalid rank. >>>>>>> >>>>>>> As discussed in the section about mpiexec, an implementation may decide >>>>>> to provide a soft process count argument. So "mpiexec -n 10 -soft 5 ./cpi" >>>>>> can start any number between 5 and 10 processes. But that does not affect >>>>>> the meaning of the size of MPI_COMM_WORLD: regardless of the number of >>>>>> processes the implementation decides to start, MPI_Comm_size will return >>>>>> the _actual_ number of processes started. >>>>>>> >>>>>>> -d >>>>>>> >>>>>>> On Sep 17, 2010, at 11:22 AM, Graham, Richard L. wrote: >>>>>>> >>>>>>>> We need to clearly define what N or M is and not leave it to the >>>>>> implementation. 100% of the codes that seen over the past 15 years that >>>>>> check this value use it to indicate how many process have started. Any >>>>>> thing else is really useless, aside from letting the user find out how many >>>>>> processes actually started up, and then know how many did not start up. >>>>>>>> >>>>>>>> Rich >>>>>>>> >>>>>>>> >>>>>>>> On 9/17/10 4:27 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: >>>>>>>> >>>>>>>> So the Run-Through Stabilization proposal has been updated per our >>>>>> discussion in the working group meeting at the MPI Forum. The changes are >>>>>> summarized below: >>>>>>>> - Add a Legacy Library Support example >>>>>>>> - Clarify new error classes >>>>>>>> - Update the MPI_Init and MPI_Finalize wording to be simpler and more >>>>>> direct. >>>>>>>> - Fix wording of group creation calls versus communicator creation >>>>>> calls. >>>>>>>> >>>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>>> web/wiki/ft/run_through_stabilization >>>>>>>> >>>>>>>> >>>>>>>> One question that we discussed quite a bit during the meeting was the >>>>>> issue of the return value of MPI_Comm_size() when processes fail during >>>>>> launch. I attempted to capture the discussion in the room in the Open >>>>>> Question attached to the discussion of MPI_Init: >>>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>>> web/wiki/ft/run_through_stabilization#MPI_INIT >>>>>>>> >>>>>>>> Open question: >>>>>>>> If the user asks to start N processes on the command line, and only M >>>>>> processes were successfully launched (where M < N), then what should be >>>>>> returned from MPI_COMM_SIZE? >>>>>>>> >>>>>>>> The return value must be consistent across all alive members of the >>>>>> group. The issue is if it should return N or M. >>>>>>>> >>>>>>>> The feeling in the room was that since the MPI standard does not define >>>>>> the ability for the user to ask for a specific number of processes before >>>>>> initthen it is hard to define that this is the number it should be. >>>>>>>> >>>>>>>> So it is left to the implementation whether it is M or N. If it is M, >>>>>> then the user has other techniques to find out what it originally asked for >>>>>> (e.g., by passing it as a command line argument to the application itself). >>>>>>>> >>>>>>>> >>>>>>>> What do people think about the MPI_Comm_size issue? >>>>>>>> >>>>>>>> -- Josh >>>>>>>> >>>>>>>> ------------------------------------ >>>>>>>> Joshua Hursey >>>>>>>> Postdoctoral Research Associate >>>>>>>> Oak Ridge National Laboratory >>>>>>>> http://BLOCKEDBLOCKEDwww.BLOCKEDBLOCKEDcs.indiana.edu/~jjhursey >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> >>> ------------------------------------ >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://www.cs.indiana.edu/~jjhursey >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question
by Graham, Richard L. 20 Sep '10

20 Sep '10

I mean that if we don't clearly define what the mpi_comm_size() returns and have it be standardized, it then becomes something that users can't depend on, since it is implementation specific. Not suggesting standardizing on the command line parameters - we don't now, and still all understand the same thing that comm_size returns. I am suggesting we standardize what these do, and if we need more data than we have now, add the capability to ask for that information. Rich On 9/20/10 6:59 PM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: Sorry, I didn't understand your last sentence. What did you mean by "pick your meaning of a return value"? I might have missed that in a previous email. Are you advocating standardizing the command line parameters (-n and -soft)? Or a way to query them? -d On Sep 20, 2010, at 1:28 PM, Graham, Richard L. wrote: > My objection is not defining what the value of mpi_comm_size() returns. I think that it should return a well define value, and I would advocate for the number of ranks that have been started, which is what it returns today. If we want to be able to have users ask about command line options - soft limits, hard limits, or ?, we should provide a standard way to ask this question. I may have missed what you are proposing, but "pick your meaning of a return value" seems like it would lead to non-portable apps. > > Rich > > > On 9/20/10 10:51 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: > > How would it be specified? > > "To start an MPI application the user should call "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or will fail with error code XXX." > > This would make any runtime that provides the -soft parameter noncompliant. Do we really want to do this? > > Alternatively, we could define it something like: > > "To start an MPI application the user should cal "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or another number or MPI processes depending on other parameters specified by the user, or fail with an error code." > > But isn't this what we have now? What's wrong with that? > > -d > > On Sep 20, 2010, at 9:39 AM, Graham, Richard L. wrote: > >> I strongly disagree here. This is such a basic bit of data, we clearly need to specify the output. >> >> Rich >> >> ----- Original Message ----- >> From: Joshua Hursey [mailto:[email protected]] >> Sent: Monday, September 20, 2010 10:09 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group <mpi3-ft(a)lists.mpi-forum.org> >> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >> >> This is an interesting discussion (I am glad it is archived). >> >> I feel that Darius makes a strong argument about leaving it undefined in the standard for many of the reasons that have been mentioned. I think an implementation can decide then if one way is more important to their users than another. It is my opinion that defining it does not gain us much since most applications base their initialization on the result of MPI_Comm_size() instead of requiring a specific number of processes. Further, if they do require a specific number, they can interpret the results of MPI_Comm_size() and choose to abort or work around the difference. >> >> -- Josh >> >> On Sep 17, 2010, at 11:00 AM, Graham, Richard L. wrote: >> >>> Why would we select one behaviour over another ? Right now users use the value returned from mpi_comm_size() to determine how to setup their parallel job, so telling them what the actually got seems like the most obvious return value to have, especially if we do not provide a way to get both what they asked for, and what they actually got. Having this be implementation specific does not seem to be prudent, since they would have to know what implementation they are using to figure out what to do with the data. I believe this is very different than an implementation choice to use an eager protocol for large message vs. a rendezvous which does not have correctness implications. >>> >>> Also, at startup the user has not set anything up yet, with respect to MPI, so forming a dense communicator is well defined and the easiest thing to do. Once failure occurs, I will argue exactly the opposite, based on the outcome of such a decisions, and the fact that we will provide them a way within MPI to figure out what has happened. >>> >>> Rich >>> >>> On 9/17/10 10:14 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: >>> >>> >>> >>> I don't think we need to choose one or the other (in fact I feel strongly that we should not force one behavior or the other). The choice to have MPI_COMM_WORLD contain N or M processes (or failing if it can't get all N) is implementation dependent. Presumably the behavior would be user selectable (e.g., using a -soft option or something similar to mpiexec). >>> >>> The user would use the mechanisms we will provide to deal with any dead processes (e.g., validating a communicator, etc). >>> >>> -d >>> >>> On Sep 17, 2010, at 4:02 PM, Bronevetsky, Greg wrote: >>> >>>> I agree that MPI_Comm_size should return the number of ranks in the communicator regardless of whether they're operational or not. However, this just pushes the question further back: if the user asked for N processes but only M have started up, how many ranks should MPI_COMM_WORLD have? Either choice is going to be self-consistent from MPI's point of view. If its N, then some ranks will be dead. If it is M then the application may not have enough processes to work with. The former (N) case has property that the user doesn't need to add code to check for this condition since their existing error checking code will catch this situation. The latter case (M) is nice because it cheaper to check whether we got fewer processes than expected than to explicitly try to communicate with them. >>>> >>>> As such, I don't see a strong motivation for choosing either. However, we should just pick one and stick with it to avoid unnecessary API divergence. >>>> >>>> Greg Bronevetsky >>>> Lawrence Livermore National Lab >>>> (925) 424-5756 >>>> bronevetsky(a)llnl.gov >>>> http://greg.bronevetsky.com >>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Bronis R. de Supinski >>>>> Sent: Friday, September 17, 2010 5:23 AM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >>>>> >>>>> >>>>> I agree with Rich and Darius. >>>>> >>>>> On Fri, 17 Sep 2010, Darius Buntinas wrote: >>>>> >>>>>> >>>>>> I don't think we should change the standard in this case. For >>>>> MPI_Comm_size to have any useful meaning, it needs to return the size of >>>>> the communicator: i.e., if comm_size returns N, you should be able to do a >>>>> send to processes 0 through N-1. Of course if some of those processes have >>>>> failed, you'll get an error associated with the process failure, but never >>>>> an error for an invalid rank. >>>>>> >>>>>> As discussed in the section about mpiexec, an implementation may decide >>>>> to provide a soft process count argument. So "mpiexec -n 10 -soft 5 ./cpi" >>>>> can start any number between 5 and 10 processes. But that does not affect >>>>> the meaning of the size of MPI_COMM_WORLD: regardless of the number of >>>>> processes the implementation decides to start, MPI_Comm_size will return >>>>> the _actual_ number of processes started. >>>>>> >>>>>> -d >>>>>> >>>>>> On Sep 17, 2010, at 11:22 AM, Graham, Richard L. wrote: >>>>>> >>>>>>> We need to clearly define what N or M is and not leave it to the >>>>> implementation. 100% of the codes that seen over the past 15 years that >>>>> check this value use it to indicate how many process have started. Any >>>>> thing else is really useless, aside from letting the user find out how many >>>>> processes actually started up, and then know how many did not start up. >>>>>>> >>>>>>> Rich >>>>>>> >>>>>>> >>>>>>> On 9/17/10 4:27 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: >>>>>>> >>>>>>> So the Run-Through Stabilization proposal has been updated per our >>>>> discussion in the working group meeting at the MPI Forum. The changes are >>>>> summarized below: >>>>>>> - Add a Legacy Library Support example >>>>>>> - Clarify new error classes >>>>>>> - Update the MPI_Init and MPI_Finalize wording to be simpler and more >>>>> direct. >>>>>>> - Fix wording of group creation calls versus communicator creation >>>>> calls. >>>>>>> >>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>> web/wiki/ft/run_through_stabilization >>>>>>> >>>>>>> >>>>>>> One question that we discussed quite a bit during the meeting was the >>>>> issue of the return value of MPI_Comm_size() when processes fail during >>>>> launch. I attempted to capture the discussion in the room in the Open >>>>> Question attached to the discussion of MPI_Init: >>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>> web/wiki/ft/run_through_stabilization#MPI_INIT >>>>>>> >>>>>>> Open question: >>>>>>> If the user asks to start N processes on the command line, and only M >>>>> processes were successfully launched (where M < N), then what should be >>>>> returned from MPI_COMM_SIZE? >>>>>>> >>>>>>> The return value must be consistent across all alive members of the >>>>> group. The issue is if it should return N or M. >>>>>>> >>>>>>> The feeling in the room was that since the MPI standard does not define >>>>> the ability for the user to ask for a specific number of processes before >>>>> initthen it is hard to define that this is the number it should be. >>>>>>> >>>>>>> So it is left to the implementation whether it is M or N. If it is M, >>>>> then the user has other techniques to find out what it originally asked for >>>>> (e.g., by passing it as a command line argument to the application itself). >>>>>>> >>>>>>> >>>>>>> What do people think about the MPI_Comm_size issue? >>>>>>> >>>>>>> -- Josh >>>>>>> >>>>>>> ------------------------------------ >>>>>>> Joshua Hursey >>>>>>> Postdoctoral Research Associate >>>>>>> Oak Ridge National Laboratory >>>>>>> http://BLOCKEDBLOCKEDwww.BLOCKEDBLOCKEDcs.indiana.edu/~jjhursey >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://www.cs.indiana.edu/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question
by Darius Buntinas 20 Sep '10

20 Sep '10

Sorry, I didn't understand your last sentence. What did you mean by "pick your meaning of a return value"? I might have missed that in a previous email. Are you advocating standardizing the command line parameters (-n and -soft)? Or a way to query them? -d On Sep 20, 2010, at 1:28 PM, Graham, Richard L. wrote: > My objection is not defining what the value of mpi_comm_size() returns. I think that it should return a well define value, and I would advocate for the number of ranks that have been started, which is what it returns today. If we want to be able to have users ask about command line options - soft limits, hard limits, or ?, we should provide a standard way to ask this question. I may have missed what you are proposing, but "pick your meaning of a return value" seems like it would lead to non-portable apps. > > Rich > > > On 9/20/10 10:51 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: > > How would it be specified? > > "To start an MPI application the user should call "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or will fail with error code XXX." > > This would make any runtime that provides the -soft parameter noncompliant. Do we really want to do this? > > Alternatively, we could define it something like: > > "To start an MPI application the user should cal "mpiexec" or another command, which shall either start the number of MPI processes that the user requested, or another number or MPI processes depending on other parameters specified by the user, or fail with an error code." > > But isn't this what we have now? What's wrong with that? > > -d > > On Sep 20, 2010, at 9:39 AM, Graham, Richard L. wrote: > >> I strongly disagree here. This is such a basic bit of data, we clearly need to specify the output. >> >> Rich >> >> ----- Original Message ----- >> From: Joshua Hursey [mailto:[email protected]] >> Sent: Monday, September 20, 2010 10:09 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group <mpi3-ft(a)lists.mpi-forum.org> >> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >> >> This is an interesting discussion (I am glad it is archived). >> >> I feel that Darius makes a strong argument about leaving it undefined in the standard for many of the reasons that have been mentioned. I think an implementation can decide then if one way is more important to their users than another. It is my opinion that defining it does not gain us much since most applications base their initialization on the result of MPI_Comm_size() instead of requiring a specific number of processes. Further, if they do require a specific number, they can interpret the results of MPI_Comm_size() and choose to abort or work around the difference. >> >> -- Josh >> >> On Sep 17, 2010, at 11:00 AM, Graham, Richard L. wrote: >> >>> Why would we select one behaviour over another ? Right now users use the value returned from mpi_comm_size() to determine how to setup their parallel job, so telling them what the actually got seems like the most obvious return value to have, especially if we do not provide a way to get both what they asked for, and what they actually got. Having this be implementation specific does not seem to be prudent, since they would have to know what implementation they are using to figure out what to do with the data. I believe this is very different than an implementation choice to use an eager protocol for large message vs. a rendezvous which does not have correctness implications. >>> >>> Also, at startup the user has not set anything up yet, with respect to MPI, so forming a dense communicator is well defined and the easiest thing to do. Once failure occurs, I will argue exactly the opposite, based on the outcome of such a decisions, and the fact that we will provide them a way within MPI to figure out what has happened. >>> >>> Rich >>> >>> On 9/17/10 10:14 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: >>> >>> >>> >>> I don't think we need to choose one or the other (in fact I feel strongly that we should not force one behavior or the other). The choice to have MPI_COMM_WORLD contain N or M processes (or failing if it can't get all N) is implementation dependent. Presumably the behavior would be user selectable (e.g., using a -soft option or something similar to mpiexec). >>> >>> The user would use the mechanisms we will provide to deal with any dead processes (e.g., validating a communicator, etc). >>> >>> -d >>> >>> On Sep 17, 2010, at 4:02 PM, Bronevetsky, Greg wrote: >>> >>>> I agree that MPI_Comm_size should return the number of ranks in the communicator regardless of whether they're operational or not. However, this just pushes the question further back: if the user asked for N processes but only M have started up, how many ranks should MPI_COMM_WORLD have? Either choice is going to be self-consistent from MPI's point of view. If its N, then some ranks will be dead. If it is M then the application may not have enough processes to work with. The former (N) case has property that the user doesn't need to add code to check for this condition since their existing error checking code will catch this situation. The latter case (M) is nice because it cheaper to check whether we got fewer processes than expected than to explicitly try to communicate with them. >>>> >>>> As such, I don't see a strong motivation for choosing either. However, we should just pick one and stick with it to avoid unnecessary API divergence. >>>> >>>> Greg Bronevetsky >>>> Lawrence Livermore National Lab >>>> (925) 424-5756 >>>> bronevetsky(a)llnl.gov >>>> http://greg.bronevetsky.com >>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Bronis R. de Supinski >>>>> Sent: Friday, September 17, 2010 5:23 AM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question >>>>> >>>>> >>>>> I agree with Rich and Darius. >>>>> >>>>> On Fri, 17 Sep 2010, Darius Buntinas wrote: >>>>> >>>>>> >>>>>> I don't think we should change the standard in this case. For >>>>> MPI_Comm_size to have any useful meaning, it needs to return the size of >>>>> the communicator: i.e., if comm_size returns N, you should be able to do a >>>>> send to processes 0 through N-1. Of course if some of those processes have >>>>> failed, you'll get an error associated with the process failure, but never >>>>> an error for an invalid rank. >>>>>> >>>>>> As discussed in the section about mpiexec, an implementation may decide >>>>> to provide a soft process count argument. So "mpiexec -n 10 -soft 5 ./cpi" >>>>> can start any number between 5 and 10 processes. But that does not affect >>>>> the meaning of the size of MPI_COMM_WORLD: regardless of the number of >>>>> processes the implementation decides to start, MPI_Comm_size will return >>>>> the _actual_ number of processes started. >>>>>> >>>>>> -d >>>>>> >>>>>> On Sep 17, 2010, at 11:22 AM, Graham, Richard L. wrote: >>>>>> >>>>>>> We need to clearly define what N or M is and not leave it to the >>>>> implementation. 100% of the codes that seen over the past 15 years that >>>>> check this value use it to indicate how many process have started. Any >>>>> thing else is really useless, aside from letting the user find out how many >>>>> processes actually started up, and then know how many did not start up. >>>>>>> >>>>>>> Rich >>>>>>> >>>>>>> >>>>>>> On 9/17/10 4:27 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote: >>>>>>> >>>>>>> So the Run-Through Stabilization proposal has been updated per our >>>>> discussion in the working group meeting at the MPI Forum. The changes are >>>>> summarized below: >>>>>>> - Add a Legacy Library Support example >>>>>>> - Clarify new error classes >>>>>>> - Update the MPI_Init and MPI_Finalize wording to be simpler and more >>>>> direct. >>>>>>> - Fix wording of group creation calls versus communicator creation >>>>> calls. >>>>>>> >>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>> web/wiki/ft/run_through_stabilization >>>>>>> >>>>>>> >>>>>>> One question that we discussed quite a bit during the meeting was the >>>>> issue of the return value of MPI_Comm_size() when processes fail during >>>>> launch. I attempted to capture the discussion in the room in the Open >>>>> Question attached to the discussion of MPI_Init: >>>>>>> https://BLOCKEDBLOCKEDsvn.mpi-forum.org/trac/mpi-forum- >>>>> web/wiki/ft/run_through_stabilization#MPI_INIT >>>>>>> >>>>>>> Open question: >>>>>>> If the user asks to start N processes on the command line, and only M >>>>> processes were successfully launched (where M < N), then what should be >>>>> returned from MPI_COMM_SIZE? >>>>>>> >>>>>>> The return value must be consistent across all alive members of the >>>>> group. The issue is if it should return N or M. >>>>>>> >>>>>>> The feeling in the room was that since the MPI standard does not define >>>>> the ability for the user to ask for a specific number of processes before >>>>> initthen it is hard to define that this is the number it should be. >>>>>>> >>>>>>> So it is left to the implementation whether it is M or N. If it is M, >>>>> then the user has other techniques to find out what it originally asked for >>>>> (e.g., by passing it as a command line argument to the application itself). >>>>>>> >>>>>>> >>>>>>> What do people think about the MPI_Comm_size issue? >>>>>>> >>>>>>> -- Josh >>>>>>> >>>>>>> ------------------------------------ >>>>>>> Joshua Hursey >>>>>>> Postdoctoral Research Associate >>>>>>> Oak Ridge National Laboratory >>>>>>> http://BLOCKEDBLOCKEDwww.BLOCKEDBLOCKEDcs.indiana.edu/~jjhursey >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://BLOCKEDBLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://www.cs.indiana.edu/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Stabilization Updated & MPI_Comm_size question
by Richard Treumann 20 Sep '10

20 Sep '10

The MPI Standard currently states that communicator creation is collective and deterministic. (I did not go back and read the wording so I may be missing some nuance). To me that means that when a communicator handle is returned, every process has certain pieces of information encoded in the opaque communicator object that it can assume match at every member. This shared view includes the number of processes negotiated and an agreement on which process corresponds to each rank. It does not mean that every process that was intended to be in the communicator is still alive and reachable. Are you changing the rule that "communicator creation is collective and deterministic"? If not, there are things which can occur before the "collective and deterministic" commit point and things that occur after. It seems to me that MPI_Comm_size must return the number that was committed and if you need to find how many are left after post-commit problems, it demands a different call. If you need to find out how many were desired when fewer may have been committed, that too is another call. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363

1 0

Re: [Mpi3-ft] Defining the state of MPI after an error
by Richard Treumann 20 Sep '10

20 Sep '10

Josh Did you see this? "For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. " If my MPI implementation is going to tell you it "allows for correct operation after returning that error class", the standard needs to tell me which behavior is "correct operation" and what is not "correct operation". As I mentioned to Bronis, I have no problem with an advise to implementors urging that they at least allow an option for continued use of libmpi, after an error returns, but at the users own risk. Offhand, I cannot see any harm in urging MPI implementors to promise that all locks will be released when a subroutine returns non-SUCCESS. Can you give me a use case for the added complexity of CAN_CONTINUE vs CANNOT_CONTINUE.? Consider that an error class of CAN_CONTINUE that is returned because the MPI implementor does not really know what "correct operation" means and decides to wing it is not very useful. An MPI implementor that can get in real trouble for negligence would need to be very cautious about returning CAN_CONTINUE. Just asking the MPI implementor to refrain from flagging the error "MPI disabled after prior error" on otherwise correct future calls seems as good to me. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Joshua Hursey <jjhursey(a)open-mpi.org> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 09/20/2010 01:51 PM Subject: Re: [Mpi3-ft] Defining the state of MPI after an error Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org So the proposal makes no requirements about the state of the distributed environment after an error. All it defines is a error class to be returned by an MPI implementation once the MPI implementation can no longer function correctly. This provides a necessary feedback mechanism for the application to determine if and how the MPI implementation can be used after an error occurs. It is the responsibility of the application to avoid deadlocks and other such issues that can result from handling and recovering from errors. If an application is designed to recover from MPI_ERR_TAG that's great. If the MPI library allows for correct operation after returning that error class, then even better. If the MPI library cannot continue operation after that error then it can block subsequent operations by returning MPI_ERR_CANNOT_CONTINUE. I disagree with your assessment that this will be difficult to implement/test since a trivial implementation of this proposal is to set a global variable when an error occurs to always return MPI_ERR_CANNOT_CONTINUE when the application calls into the MPI library (it is a similar check as the 'is_mpi_initialized' check that has to be there anyway). If an implementation wants to do more (and it is definitely not required to do so) then it can define that in it's documentation. If an application wants to try to use MPI after an error it must understand that the error is local in nature (it cannot assume that every process received an error). If it can figure out how to recover from it, and the MPI implementation is able to function correctly afterward then we should let them figure it out. This allows us to define the boundaries of correct operation after an error. Before the application -could- keep using the MPI library after an error, but it was entirely undefined and not-portable what would happen. Now the application can portably attempt to use the MPI library after an error and know that it can expect either normal functionality (for those few implementations that do more than the minimum necessary) or MPI_ERR_CANNOT_CONTINUE in which the library locks them out and they then know to terminate normally. I hope this helps a bit, but maybe I am missing the core problem that you are trying to get at. -- Josh On Sep 20, 2010, at 1:09 PM, Bronis R. de Supinski wrote: > > Dick: > > Re: >> I did not intend to ignore your use case. > > No problem. > >> I did mention that I have no worries about asking MPI implementations >> to refrain from blocking future MPI calls after an error is detected. >> That was an implicit recognition of your use case. > > OK, that helps. > >> The MPI standard already forbids having an MPI call on one thread block >> progress on other threads. I would interpret that to include a case >> where a thread is blocked in a collective communication or a MPI_Recv >> that will never be satisfied. That is, the blocked MPI call cannot >> prevent other threads from using libmpi. Requiring libmpi to release >> any lock it took even when doing an error return would be logical but >> may not be implied by what is currently written. > > The current text provides no such guarantee. Once anerror is > returned anywhere, all bets are off (at least that is how I > have read it; I would need to go back through the text to > find the exact words that cause my concern). > >> Communicators provide a sort of isolation that keeps stray crap from >> failed operations from spilling over (such as eager sent message for >> which the MPI_Recv failed). If the tool uses its own threads and >> private communicators, I agree it is reasonable to ask any libmpi to >> avoid sabotaging that communication. > > That would be perfect from my perspective. > >> Where I get concerned is when we start talking about affirmative >> requirements for distributed MPI state after an error > > I don't think we can have those beyond "best effort". > The errors may indicate problems that make further > communication impossible -- perhaps because of the > erroneous action or just due to the state of the > network or other processes. I do think we can require > accurate return values and have an advice to implementers > that suggests best effort following errors. I believe > that would satisfy my requirements. > > Bronis > > > > >> >> Dick >> >> Dick Treumann - MPI Team >> IBM Systems & Technology Group >> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >> Tele (845) 433-7846 Fax (845) 433-8363 >> >> >> >> From: "Bronis R. de Supinski" <bronis(a)llnl.gov> >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >> Date: 09/20/2010 12:46 PM >> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error >> Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org >> >> ________________________________ >> >> >> >> >> Dick: >> >> You seem to be ignoring my use case. Specifically, I >> have tool threads that use MPI. Their use of MPI should >> be unaffected by all of the scenarios that you are raising. >> However, the standard provides no way for me to tell if >> they work correctly in these situations. I just have to >> cross my fingers and hope. >> >> FYI: Your implementation has long met this requirement >> (my hopes are not dashed with it). Others have begun to >> recently. In any event, I would like some way to tell... >> >> Further, it is useful in many other scenarios apply to know >> that the implementation intends to remain usable. I am not >> looking for a promise of correct execution; I am looking >> for a promise of best effort and accurate return codes. >> >> Bronis >> >> >> >> On Mon, 20 Sep 2010, Richard Treumann wrote: >> >>> >>> If there is any question about whether these calls are still valid after an error with an error handler that returns (MPI_ERRORS_RETURN or user handler) >>> >>> MPI_Abort, >>> MPI_Error_string >>> MPI_Error_class >>> >>> I assume it should be corrected as a trivial oversight in the original text. >>> >>> I would regard the real issue as being the difficulty with assuring the state of remote processes. >>> >>> There is huge difficulty in making any promise about how an interaction between a process that has not taken an error and one that has will behave. >>> >>> For example, if there were a loop of 100 MPI_Bcast calls and on iteration 5, rank 3 uses a bad communicator, what is the proper state? Either a sequence number is mandated so the other ranks hang quickly or a sequence number is prohibited so everybody keeps going until the "end" when the missing MPI_Bcast becomes critical. Of course, with no sequence number, some tasks are stupidly using the iteration n-1 data for their iteration n computation. >>> >>> >>> >>> >>> >>> >>> Dick Treumann - MPI Team >>> IBM Systems & Technology Group >>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 >>> Tele (845) 433-7846 Fax (845) 433-8363 >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0