February 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] run through stabilization user-guide
by Darius Buntinas 08 Feb '11

08 Feb '11

I believe the standard does define process failures as fail-stop. However, the standard does not describe how to detect failures. Doing this would severely limit implementations. I suspect that most implementations will rely on external mechanisms to detect process failure, such as ipmi, or a ras system. MPICH2 relies on the hydra process manager to detect abnormal process termination and node failures. I'm not sure whether fail-stop is exactly the same thing as abnormal process termination, but realistically this is how most vendors will implement it. Distinguishing between unresponsive processes and crashed processes is impossible to do. Even you add heartbeat messages between MPI libraries will only tell you that the library hasn't crashed. You could construct a scenario where the application is in an endless loop but doesn't call MPI_Recv, and the sender runs out of resources, even though the library is still alive. I believe crashed, but not terminated, would not be considered fail-stopped (but probably byzantine), and so outside the scope of the standard. Users that need to handle these kinds of failures would need to implement their own mechanism to detect hung processes, since they are in the best position to decide when a process is hung. -d On Feb 8, 2011, at 5:14 AM, Toon Knapen wrote: > > The spec explicitly doesnï¿½t define the meaning of failure because it is a very low-level concept. The MPI implementation is responsible for detecting failures and defining what qualifies as one. The only guarantee that you can rely on is that if the ï¿½failureï¿½ is bad enough that a given MPI rank canï¿½t communicate with others, then MPI will have to either abort the application completely or eventually report this failure via the FT API. > > > I understand it is really hard to define what is exactly a failure. But if the future standard does not describe it, is'nt there a risk that apps that rely on the FT API will become hard to port from one implementation to another ? > > For instance, suppose I have one process constantly sending messages to the other one. Suppose the second process is dead, the first process thus might either detect that the second process is down and inform the app. Or the first process might not tell anything to the app and buffer all messages to be send. But this buffer might grow so big that the first process will run out-of-memory and dies too. > > So I think it would be usefull that the mpi-library is given some bounds within which it should function and if it is not able to function within these bounds it should raise an error. > > For instance a simple limit might be that each node should respond to a ping within 1s (or 1ms. or ...). If a node can not be pinged (within this limit), the node is considered to be dead. > > Another bound that might be specified and that would serve my example above is e.g. the memory bounds within which MPI should function. For instance, the app (or user) might decide to allocate 100 MB of (buffer-) memory to MPI. If the app fills this buffer and the MPI-lib is not able to flush the data sufficiently fast to the other nodes, an error will be raised. At that point the app is aware that there is a failure to communicate and can take appropriate action: slow down a bit with the sending or consider the other node to have failed. > > In the above, it is the MPI lib that always detects the error. However if the MPI library does not guarantee what a failure is exactly, I might have to detect failures in the app (to be independent of the free interpretation of failure). In that case I might e.g. need to always do non-blocking sends and recvs (to avoid being blocked until eternity in case of a failure) and see if the messages arrive in a timely manner, if not I consider the other process to be dead. In that case however, I would also need functionality to tell to the MPI-lib that a specific (comm,rank) should be considered MPI_RANK_STATE_NULL. > > I'm sorry if the above has been discussed at length already and the forum already decided not to define 'failure' (I should'nt have left the MPI-scene for the last three years, what was I thinking ;-). Trying to define 'failure' might be opening pandora's box but I'm looking at this from the appication point-of-view. IMHO FT is either just about 'going down gracefully' or about 'trying to finish the job'. > > > Note that this view is focused on process failures. When it is applied to things like network partitions (this includes the case you mentioned where one process canï¿½t talk to any other due to a failed network card) then processes on both parts of the partition may be informed that the others have failed. As such, when connectivity is restored, since MPI will be responsible to maintaining self-consistency of its previous notifications, itï¿½ll have to kill processes on one side of the partition to keep consistent with the notifications it gave to the other partition. > > > Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed. > > toon > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Toon Knapen 08 Feb '11

08 Feb '11

> > The spec explicitly doesn’t define the meaning of failure because it is > a very low-level concept. The MPI implementation is responsible for > detecting failures and defining what qualifies as one. The only guarantee > that you can rely on is that if the “failure” is bad enough that a given MPI > rank can’t communicate with others, then MPI will have to either abort the > application completely or eventually report this failure via the FT API. > I understand it is really hard to define what is exactly a failure. But if the future standard does not describe it, is'nt there a risk that apps that rely on the FT API will become hard to port from one implementation to another ? For instance, suppose I have one process constantly sending messages to the other one. Suppose the second process is dead, the first process thus might either detect that the second process is down and inform the app. Or the first process might not tell anything to the app and buffer all messages to be send. But this buffer might grow so big that the first process will run out-of-memory and dies too. So I think it would be usefull that the mpi-library is given some bounds within which it should function and if it is not able to function within these bounds it should raise an error. For instance a simple limit might be that each node should respond to a ping within 1s (or 1ms. or ...). If a node can not be pinged (within this limit), the node is considered to be dead. Another bound that might be specified and that would serve my example above is e.g. the memory bounds within which MPI should function. For instance, the app (or user) might decide to allocate 100 MB of (buffer-) memory to MPI. If the app fills this buffer and the MPI-lib is not able to flush the data sufficiently fast to the other nodes, an error will be raised. At that point the app is aware that there is a failure to communicate and can take appropriate action: slow down a bit with the sending or consider the other node to have failed. In the above, it is the MPI lib that always detects the error. However if the MPI library does not guarantee what a failure is exactly, I might have to detect failures in the app (to be independent of the free interpretation of failure). In that case I might e.g. need to always do non-blocking sends and recvs (to avoid being blocked until eternity in case of a failure) and see if the messages arrive in a timely manner, if not I consider the other process to be dead. In that case however, I would also need functionality to tell to the MPI-lib that a specific (comm,rank) should be considered MPI_RANK_STATE_NULL. I'm sorry if the above has been discussed at length already and the forum already decided not to define 'failure' (I should'nt have left the MPI-scene for the last three years, what was I thinking ;-). Trying to define 'failure' might be opening pandora's box but I'm looking at this from the appication point-of-view. IMHO FT is either just about 'going down gracefully' or about 'trying to finish the job'. > > > Note that this view is focused on process failures. When it is applied to > things like network partitions (this includes the case you mentioned where > one process can’t talk to any other due to a failed network card) then > processes on both parts of the partition may be informed that the others > have failed. As such, when connectivity is restored, since MPI will be > responsible to maintaining self-consistency of its previous notifications, > it’ll have to kill processes on one side of the partition to keep consistent > with the notifications it gave to the other partition. > Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed. toon

1 0

Re: [Mpi3-ft] Run-Through Stabilization Users Guide
by Bronevetsky, Greg 07 Feb '11

07 Feb '11

> I think we should steer clear of even hinting to the application that we > are going to kill processes so that we can save memory. Now the process > could fail because MPI tried to allocate memory and ran out - at which > point the MPI should return an error, and shutdown services - Maybe a good > use case for CANNOT_CONTINUE. That situation is slightly different that > proactively killing processes because we don't want to/cannot track their > state. That sounds like a sure fire way to make users flee from the MPI to > something else. > In that case, how do we deal with network partitions. As I mentioned in my email to Toon, the current semantics effectively force us to kill all the processes on one side of a network partition once the partition is healed. The same is true for any process that is erroneously judged to have failed because of a transient network problem. What do we do then? > As far as the application is concerned, it only needs to allocate an array > of MPI_Rank_info objects the size of that retuned by MPI_Comm_validate_all. > Which should be small, but is inclusive of all the known failures > (recognized and unrecognized). > > We could add a flag to the local MPI_Comm_validate that would return a list > of the unrecognized failures, instead of all failures. Or, a bit more > specifically, a 'state' key argument to the function, and it will return > only those processes that match that state. This would allow the > application to manage smaller sets of failures without having to iterate > over previously recognized failures. What do you think about that? > I like both ideas very much. The use of a flag for MPI_Comm_validate can also be used MPI_Comm_validate_all as well and it avoids the need for two collectives. Also, having MPI return a list would be a big improvement. The set of failed processes will be sparse in the overall set of MPI ranks, so forcing developers to allocate and iterate over arrays will be very expensive. Lists solve this problem and in are a generally more scalable way to communicate information about a few ranks from the set of all possible ranks. > Ah I see what you are saying now. So add one more state in order to the > taxonomy: ... > Globally cleared really only has meaning at the full communicator level > (used to determine if collectives are enabled/disabled). So wouldn't it be > better to have a way to query the communicator to see if it was globally > cleared or not, in order for the application to decide if collectives can > be used or not. We talked a while back about adding an attribute to the > communicator to indicate this, but I don't think it ever made it into the > proposal. I'll make a note about this, and maybe we can discuss what it > should look like next week. > Ah, now I see what You're saying! If any rank in a communicator is not globally cleared, then the only remedy is to clear the entire communicator. As such, the only useful API is one that checks the global-cleared status of communicators, not individual ranks. Yeah, that makes sense. Also, I think you should change the terminology from "globally" cleared to something more connected to communicators. For example, a rank may be "locally cleared" and a communicator can be "cleared" but a rank can't be "globally cleared" Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Bronevetsky, Greg 06 Feb '11

06 Feb '11

Toon, thank you for the feedback! Let me try to answer your questions, though it would also be great to hear from Josh who's now the guiding spirit of the group. The spec explicitly doesn't define the meaning of failure because it is a very low-level concept. The MPI implementation is responsible for detecting failures and defining what qualifies as one. The only guarantee that you can rely on is that if the "failure" is bad enough that a given MPI rank can't communicate with others, then MPI will have to either abort the application completely or eventually report this failure via the FT API. Note that this view is focused on process failures. When it is applied to things like network partitions (this includes the case you mentioned where one process can't talk to any other due to a failed network card) then processes on both parts of the partition may be informed that the others have failed. As such, when connectivity is restored, since MPI will be responsible to maintaining self-consistency of its previous notifications, it'll have to kill processes on one side of the partition to keep consistent with the notifications it gave to the other partition. We haven't talked about the process itself detecting failures independently of the MPI library. Right now the only thing a process can do is call MPI_Abort() if it notices that its core is not operating correctly. However, in the future I can easily see us providing applications with call via which they can inform MPI about network failures and failures of other processes. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Toon Knapen Sent: Sunday, February 06, 2011 10:19 AM To: mpi3-ft(a)lists.mpi-forum.org Subject: [Mpi3-ft] run through stabilization user-guide Hi all, (I've ended up on this ml because of a recent discussion on the OpenMPI ml referring to this user-guide and because I'm working on a parallel financial app that runs in batch-mode but will also be run in online-mode. The calculations are only loosely coupled calculations but stability/fault-tolerance is important when running in batch-mode and certainly when running online) So I ended up reading the user guide (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_users_guide) and since it seems destined to app developers like me, I'll provide some feedback from my pov: In the motivation section, paragraph 4: Hopefully not a silly question but what is actually considered to be a failure ? Looking into the archives of this ml there are many discussions on the topic but what is actually the current conclusion? Additionally, is it possible that a process can detect that the process itself fails, e.g. if its network went down and thus unable to communicate any further. Finally, I think there is a typo in the first example in the section Point-to-Point operators. It relies on 'status' for knowing the failed rank while IMO it should be 'peer'. toon

1 0

[Mpi3-ft] run through stabilization user-guide
by Toon Knapen 06 Feb '11

06 Feb '11

Hi all, (I've ended up on this ml because of a recent discussion on the OpenMPI ml referring to this user-guide and because I'm working on a parallel financial app that runs in batch-mode but will also be run in online-mode. The calculations are only loosely coupled calculations but stability/fault-tolerance is important when running in batch-mode and certainly when running online) So I ended up reading the user guide ( https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_users_guide) and since it seems destined to app developers like me, I'll provide some feedback from my pov: In the motivation section, paragraph 4: Hopefully not a silly question but what is actually considered to be a failure ? Looking into the archives of this ml there are many discussions on the topic but what is actually the current conclusion? Additionally, is it possible that a process can detect that the process itself fails, e.g. if its network went down and thus unable to communicate any further. Finally, I think there is a typo in the first example in the section Point-to-Point operators. It relies on 'status' for knowing the failed rank while IMO it should be 'peer'. toon

1 0

Re: [Mpi3-ft] Exit Code from 'mpirun' upon failure recovery
by Joshua Hursey 04 Feb '11

04 Feb '11

On Feb 4, 2011, at 12:36 PM, Darius Buntinas wrote: > > Here's a first stab: > > Without faults, it would be reasonable for mpiexec to return the exit code of rank 0. If we extend this to the case with faults and fault tolerant programs, we could say that mpiexec returns the return code from the lowest ranked process that exits after calling MPI_Finalize. If there is no such process, the return code specified in the last call to MPI_Abort(). If no process calls MPI_Finalize or MPI_Abort, return the return code of rank 0. > > This way if the process chooses to handle faults and continues the non-failed processes will eventually call MPI_Finalize. These processes can then decide what return code to return, since we have to choose one, let's choose the lowest ranked one. > > MPI_Abort can specify a communicator other than MPI_COMM_WORLD, and a high-quality implementation would abort only the processes in that communicator, leaving the remaining processes to continue. In this case we don't want to save the return code of the MPI_Abort, since the app is still running. If these processes eventually call MPI_Finalize, that's covered by the first case. But if these processes run into another problem and call MPI_Abort, we should return the return code from that MPI_Abort. So only the last MPI_Abort should count. Of course 'last' may be subject to some nondeterminism, but that should be OK. > > The last case is a catch-all where processes just die. So we might as well return the return code of rank 0. > > What do you think? Of course this would be 'advice to implementors' and not required. I think this is a pretty reasonable definition. If others agree, I can add it to the proposal since I think it would make a pretty useful 'advice to implementors'. On a side note, in Open MPI, we have had a variety of requests for behaviors in similar scenarios (e.g., return the lowest, non-zero value from any rank, highest rank, lowest rank, a specific rank, and many others). So having something defined in the standard would go a long way to helping sort through these user requests. Thanks, Josh > > -d > > > > On Feb 4, 2011, at 10:50 AM, Joshua Hursey wrote: > >> The standard is pretty cagey about this issue. The only place where I see it referenced is after MPI_Abort() where it says: >> --------------- >> Advice to users. Whether the errorcode is returned from the executable or from the MPI process startup mechanism (e.g., mpiexec), is an aspect of quality of the MPI library but not mandatory. (End of advice to users.) >> >> Advice to implementors. Where possible, a high-quality implementation will try to return the errorcode from the MPI process startup mechanism (e.g. mpiexec or singleton init). (End of advice to implementors.) >> --------------- >> >> So it does not say what happens if MPI_Abort is called from multiple processes each with different errorcodes, or if the MPI implementation chooses to continue with the application and it completes normally later. >> >> >> Since the standard is reluctant to provide us guidance, I suspect that any decision we make is not appropriate for explicit standardization -- Maybe it can be an additional 'Advice to implementors'. I was mostly trying to see if anyone had strong feelings on the expected behavior in the various fault tolerant scenarios. In particular, I know that some applications check for non-zero return code from 'mpirun' as one indicator that the job did not complete successfully, and determine if they should take recovery steps after job completion. >> >> -- Josh >> >> On Feb 4, 2011, at 11:12 AM, Darius Buntinas wrote: >> >>> >>> What does the standard suggest for the case when different processes return different return codes? Can't we use the same approach? >>> >>> -d >>> >>> On Feb 4, 2011, at 8:19 AM, Joshua Hursey wrote: >>> >>>> So this is not really appropriate for the MPI standard language, but more of a user experience question. In fact this is a much larger question that implementations have to struggle with already. >>>> >>>> If a process fails in the application (either by external causes or by calling MPI_Abort), what should 'mpirun' return as its exit status? >>>> >>>> If the application intends to handle the failure and continue running after recovering then they may expect that as long as MPI_Finalize is called in all remaining processes that 'mpirun' return '0' or success. But if no process calls MPI_Finalize (because they either called MPI_Abort or terminated abnormally) that it return a non-zero value - probably one of the values that they set in MPI_Abort, if possible. Of course there is the case where the failure occurs during MPI_Finalize, to which the MPI implementation may or may not be able to act consistently depending on the timing of the failure notification. >>>> >>>> I was mucking around in this code in the Open MPI prototype, and thought I would get the opinions of the group as I move forward. >>>> >>>> Thanks, >>>> Josh >>>> >>>> ------------------------------------ >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Run-Through Stabilization: One-Sided Communications Semantics
by Joshua Hursey 04 Feb '11

04 Feb '11

I just added some semantics for the One-Sided Communications chapter. This will definitely require some further discussion, but this should get us started. The link is below: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… In addition to on-list discussion, I'll add this to our agenda for Monday. If we don't get to it, then we can add it to the next teleconf. Thanks, Josh ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Exit Code from 'mpirun' upon failure recovery
by Darius Buntinas 04 Feb '11

04 Feb '11

Here's a first stab: Without faults, it would be reasonable for mpiexec to return the exit code of rank 0. If we extend this to the case with faults and fault tolerant programs, we could say that mpiexec returns the return code from the lowest ranked process that exits after calling MPI_Finalize. If there is no such process, the return code specified in the last call to MPI_Abort(). If no process calls MPI_Finalize or MPI_Abort, return the return code of rank 0. This way if the process chooses to handle faults and continues the non-failed processes will eventually call MPI_Finalize. These processes can then decide what return code to return, since we have to choose one, let's choose the lowest ranked one. MPI_Abort can specify a communicator other than MPI_COMM_WORLD, and a high-quality implementation would abort only the processes in that communicator, leaving the remaining processes to continue. In this case we don't want to save the return code of the MPI_Abort, since the app is still running. If these processes eventually call MPI_Finalize, that's covered by the first case. But if these processes run into another problem and call MPI_Abort, we should return the return code from that MPI_Abort. So only the last MPI_Abort should count. Of course 'last' may be subject to some nondeterminism, but that should be OK. The last case is a catch-all where processes just die. So we might as well return the return code of rank 0. What do you think? Of course this would be 'advice to implementors' and not required. -d On Feb 4, 2011, at 10:50 AM, Joshua Hursey wrote: > The standard is pretty cagey about this issue. The only place where I see it referenced is after MPI_Abort() where it says: > --------------- > Advice to users. Whether the errorcode is returned from the executable or from the MPI process startup mechanism (e.g., mpiexec), is an aspect of quality of the MPI library but not mandatory. (End of advice to users.) > > Advice to implementors. Where possible, a high-quality implementation will try to return the errorcode from the MPI process startup mechanism (e.g. mpiexec or singleton init). (End of advice to implementors.) > --------------- > > So it does not say what happens if MPI_Abort is called from multiple processes each with different errorcodes, or if the MPI implementation chooses to continue with the application and it completes normally later. > > > Since the standard is reluctant to provide us guidance, I suspect that any decision we make is not appropriate for explicit standardization -- Maybe it can be an additional 'Advice to implementors'. I was mostly trying to see if anyone had strong feelings on the expected behavior in the various fault tolerant scenarios. In particular, I know that some applications check for non-zero return code from 'mpirun' as one indicator that the job did not complete successfully, and determine if they should take recovery steps after job completion. > > -- Josh > > On Feb 4, 2011, at 11:12 AM, Darius Buntinas wrote: > >> >> What does the standard suggest for the case when different processes return different return codes? Can't we use the same approach? >> >> -d >> >> On Feb 4, 2011, at 8:19 AM, Joshua Hursey wrote: >> >>> So this is not really appropriate for the MPI standard language, but more of a user experience question. In fact this is a much larger question that implementations have to struggle with already. >>> >>> If a process fails in the application (either by external causes or by calling MPI_Abort), what should 'mpirun' return as its exit status? >>> >>> If the application intends to handle the failure and continue running after recovering then they may expect that as long as MPI_Finalize is called in all remaining processes that 'mpirun' return '0' or success. But if no process calls MPI_Finalize (because they either called MPI_Abort or terminated abnormally) that it return a non-zero value - probably one of the values that they set in MPI_Abort, if possible. Of course there is the case where the failure occurs during MPI_Finalize, to which the MPI implementation may or may not be able to act consistently depending on the timing of the failure notification. >>> >>> I was mucking around in this code in the Open MPI prototype, and thought I would get the opinions of the group as I move forward. >>> >>> Thanks, >>> Josh >>> >>> ------------------------------------ >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Exit Code from 'mpirun' upon failure recovery
by Joshua Hursey 04 Feb '11

04 Feb '11

The standard is pretty cagey about this issue. The only place where I see it referenced is after MPI_Abort() where it says: --------------- Advice to users. Whether the errorcode is returned from the executable or from the MPI process startup mechanism (e.g., mpiexec), is an aspect of quality of the MPI library but not mandatory. (End of advice to users.) Advice to implementors. Where possible, a high-quality implementation will try to return the errorcode from the MPI process startup mechanism (e.g. mpiexec or singleton init). (End of advice to implementors.) --------------- So it does not say what happens if MPI_Abort is called from multiple processes each with different errorcodes, or if the MPI implementation chooses to continue with the application and it completes normally later. Since the standard is reluctant to provide us guidance, I suspect that any decision we make is not appropriate for explicit standardization -- Maybe it can be an additional 'Advice to implementors'. I was mostly trying to see if anyone had strong feelings on the expected behavior in the various fault tolerant scenarios. In particular, I know that some applications check for non-zero return code from 'mpirun' as one indicator that the job did not complete successfully, and determine if they should take recovery steps after job completion. -- Josh On Feb 4, 2011, at 11:12 AM, Darius Buntinas wrote: > > What does the standard suggest for the case when different processes return different return codes? Can't we use the same approach? > > -d > > On Feb 4, 2011, at 8:19 AM, Joshua Hursey wrote: > >> So this is not really appropriate for the MPI standard language, but more of a user experience question. In fact this is a much larger question that implementations have to struggle with already. >> >> If a process fails in the application (either by external causes or by calling MPI_Abort), what should 'mpirun' return as its exit status? >> >> If the application intends to handle the failure and continue running after recovering then they may expect that as long as MPI_Finalize is called in all remaining processes that 'mpirun' return '0' or success. But if no process calls MPI_Finalize (because they either called MPI_Abort or terminated abnormally) that it return a non-zero value - probably one of the values that they set in MPI_Abort, if possible. Of course there is the case where the failure occurs during MPI_Finalize, to which the MPI implementation may or may not be able to act consistently depending on the timing of the failure notification. >> >> I was mucking around in this code in the Open MPI prototype, and thought I would get the opinions of the group as I move forward. >> >> Thanks, >> Josh >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Exit Code from 'mpirun' upon failure recovery
by Darius Buntinas 04 Feb '11

04 Feb '11

What does the standard suggest for the case when different processes return different return codes? Can't we use the same approach? -d On Feb 4, 2011, at 8:19 AM, Joshua Hursey wrote: > So this is not really appropriate for the MPI standard language, but more of a user experience question. In fact this is a much larger question that implementations have to struggle with already. > > If a process fails in the application (either by external causes or by calling MPI_Abort), what should 'mpirun' return as its exit status? > > If the application intends to handle the failure and continue running after recovering then they may expect that as long as MPI_Finalize is called in all remaining processes that 'mpirun' return '0' or success. But if no process calls MPI_Finalize (because they either called MPI_Abort or terminated abnormally) that it return a non-zero value - probably one of the values that they set in MPI_Abort, if possible. Of course there is the case where the failure occurs during MPI_Finalize, to which the MPI implementation may or may not be able to act consistently depending on the timing of the failure notification. > > I was mucking around in this code in the Open MPI prototype, and thought I would get the opinions of the group as I move forward. > > Thanks, > Josh > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0