February 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] run through stabilization user-guide
by Joshua Hursey 10 Feb '11

10 Feb '11

I should have explained a bit more (maybe I should add this example to the users guide). By replacing the default error handler (MPI_ERRORS_ARE_FATAL) on a communicator, the application is telling the MPI library that they want to be involved in the error handling for this communicator. It is possible to specify different error handlers for different communicators, some of which may be MPI_ERRORS_ARE_FATAL. The 'fatal' nature of the error handler is scoped to the communicator that it is attached to, but may not be fatal in other communicators that overlap. An example of this (that I am calling the 'faulty subgroups' example) has multiple overlapping communicators. - MPI_COMM_WORLD: Set error handler to MPI_ERRORS_RETURN - SubComm_M: For each set of workers: Create a subcommunicator that includes the Manager and the set of workers (MPI_ERRORS_RETURN in workers, error handler function in manager) - SubComm_W: Have each set of workers create another subcommunicator that includes only the working set (excludes the manager) - set MPI_ERRORS_ARE_FATAL. So the Manager communicates with the workers using SubComm_M, and receives notification of their failure via its error handler function on SubComm_M (and on the error handler on MPI_COMM_WORLD). Workers communicate among themselves with the subcommunicator SubComm_W. If any worker fails then the SubComm_W, then the error handler of MPI_ERRORS_ARE_FATAL is triggered and -just- the subgroup is eliminated. If your application notices that a subgroup is misbehaving then you can use the MPI_Kill() command to terminate one of the worker processes in SubComm_M, which will trigger the fatal error handler in SubComm_W killing the whole subgroup of workers, leaving the manager alive and well. I should say that this is not my idea alone, but an idea that a handful of us came up with when trying to figure out how to setup a manager/worker scenario that uses the error handlers and MPI_Abort/MPI_Kill in an interesting way. So the application must opt-in to the fault tolerance semantics on a per-communicator basis, and by overlapping communicators and choosing error handlers carefully you can use MPI to help manage processes in the environment. Does that help explain the scenario? Thanks, Josh On Feb 9, 2011, at 3:14 PM, Bronevetsky, Greg wrote: > Josh, Iï¿½m rusty on the semantics here. Isnï¿½t it possible for the workers to choose MPI_ERRORS_FATAL and for the master to choose MPI_ERRORS_RETURN? > > Greg Bronevetsky > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky(a)llnl.gov > http://greg.bronevetsky.com > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Toon Knapen > Sent: Wednesday, February 09, 2011 11:42 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] run through stabilization user-guide > > > > On Wed, Feb 9, 2011 at 4:22 PM, Bronevetsky, Greg <bronevetsky1(a)llnl.gov> wrote: > > If the workers use communicators that are MPI_ERRORS_FATAL, if there is a disconnect with the master, they will be automatically aborted. Meanwhile, the master will be informed about their ï¿½failureï¿½ because of the disconnect and when connection to the physical nodes that previously hosted the aborted workers is re-established, the masterï¿½s MPI library will see that worker tasks are dead and will not need to kill the master. > > From the user guide I did not understand that there is this kind of 'interoperability' between the different error handlers. For instance the user guide says 'The application must opt-in to the fault tolerance semantics by replacing the default error handler'. > > toon > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Bronevetsky, Greg 09 Feb '11

09 Feb '11

Josh, I'm rusty on the semantics here. Isn't it possible for the workers to choose MPI_ERRORS_FATAL and for the master to choose MPI_ERRORS_RETURN? Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Toon Knapen Sent: Wednesday, February 09, 2011 11:42 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] run through stabilization user-guide On Wed, Feb 9, 2011 at 4:22 PM, Bronevetsky, Greg <bronevetsky1(a)llnl.gov<mailto:[email protected]>> wrote: If the workers use communicators that are MPI_ERRORS_FATAL, if there is a disconnect with the master, they will be automatically aborted. Meanwhile, the master will be informed about their "failure" because of the disconnect and when connection to the physical nodes that previously hosted the aborted workers is re-established, the master's MPI library will see that worker tasks are dead and will not need to kill the master. >From the user guide I did not understand that there is this kind of 'interoperability' between the different error handlers. For instance the user guide says 'The application must opt-in to the fault tolerance semantics by replacing the default error handler'. toon

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Toon Knapen 09 Feb '11

09 Feb '11

On Wed, Feb 9, 2011 at 4:22 PM, Bronevetsky, Greg <bronevetsky1(a)llnl.gov>wrote: > > > If the workers use communicators that are MPI_ERRORS_FATAL, if there is a > disconnect with the master, they will be automatically aborted. Meanwhile, > the master will be informed about their “failure” because of the disconnect > and when connection to the physical nodes that previously hosted the aborted > workers is re-established, the master’s MPI library will see that worker > tasks are dead and will not need to kill the master. > > >From the user guide I did not understand that there is this kind of 'interoperability' between the different error handlers. For instance the user guide says 'The application must opt-in to the fault tolerance semantics by replacing the default error handler'. toon

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Bronevetsky, Greg 09 Feb '11

09 Feb '11

> > Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed. You should be able to do this by creating subsets of communicators and setting error handers appropriately on the application end. Organize 'workers' into volatile groups (error handler = MPI_ERRORS_ARE_FATAL), while the 'manager' process(es) only ever participate with communicators that do not have a fatal error handler. I don't think I understand. The master and slave need to have a common communicator which will be used to communicate between the two. If the associated error handler is MPI_ERRORS_RETURN and the communication between the two is cut, the MPI lib might decide to kill either of the two ? If the workers use communicators that are MPI_ERRORS_FATAL, if there is a disconnect with the master, they will be automatically aborted. Meanwhile, the master will be informed about their "failure" because of the disconnect and when connection to the physical nodes that previously hosted the aborted workers is re-established, the master's MPI library will see that worker tasks are dead and will not need to kill the master. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Toon Knapen Sent: Wednesday, February 09, 2011 7:05 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] run through stabilization user-guide It is important to clarify what we can and cannot specify in the standard. So I appreciate your help with explaining how you think the language can be more precise or better explained. For this proposal, we are focused on graceful degradation of the job. Processes will fail(-stop), but the job as a whole is allowed the opportunity to continue operating. We are defining process failures as fail-stop failures in the standard. So we cover processes that crash, and do not continue to participate in the parallel environment. How the MPI implementation detects process failure is not defined by the standard, except by the properties that the MPI implementation must provide to the application. The MPI implementation will provide a view of the failure detector that is 'perfect' from the perspective of the application (though internally there is a fair amount of flexibility on how to provide this guarantee). I understand that processes that are detected (by the MPI lib) to fail will be stopped. That makes sense. What I do worry about is that a process or communication with a specific process is going awry. It is the responsibility of the application to design around this type of situation to ensure continued progress of their application - making a fault aware application from an otherwise fault unaware application. There are a few ways to do this, but the best solution will always be domain specific. OK It is possible that the application detects that a peer process is faulty in a different way than fail-stop (e.g., byzantine). For example, a peer process may have incurred a soft-error memory corruption, and is sending invalid data (but valid from the MPI perspective). A peer process could be checking the values, and determine that the peer is faulty. At which point it can either: - Coordinate with the other alive peers to exclude the faulty process, or - Use MPI_Comm_kill() to request that the process be terminated. MPI_Comm_kill() is not described in the user's guide, but is in the main proposal. It allows one process to kill another without killing itself (which would happen if they used MPI_Abort). Is this a scenario that you were concerned about? Yes, this is important to me. If not having a precise definition of what a 'failed' process actually is (and I understand why you avoid defining it), I'll design the app. such that it might itself decide that some (communication to a) process is failing and thus I want to be able to take the decision myself too. So this makes me feel more comfortable about it. > > Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed. You should be able to do this by creating subsets of communicators and setting error handers appropriately on the application end. Organize 'workers' into volatile groups (error handler = MPI_ERRORS_ARE_FATAL), while the 'manager' process(es) only ever participate with communicators that do not have a fatal error handler. I don't think I understand. The master and slave need to have a common communicator which will be used to communicate between the two. If the associated error handler is MPI_ERRORS_RETURN and the communication between the two is cut, the MPI lib might decide to kill either of the two ? Does that help clarify? A lot. Thanks! I think the way you (and others) clarified this, it would serve the user-guide too to discuss more in-depth what exactly a failure is, why it is not defined in detail and the ability for the app to detect errors and kill processes. toon

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Toon Knapen 09 Feb '11

09 Feb '11

> > > It is important to clarify what we can and cannot specify in the standard. > So I appreciate your help with explaining how you think the language can be > more precise or better explained. > > For this proposal, we are focused on graceful degradation of the job. > Processes will fail(-stop), but the job as a whole is allowed the > opportunity to continue operating. > > We are defining process failures as fail-stop failures in the standard. So > we cover processes that crash, and do not continue to participate in the > parallel environment. How the MPI implementation detects process failure is > not defined by the standard, except by the properties that the MPI > implementation must provide to the application. The MPI implementation will > provide a view of the failure detector that is 'perfect' from the > perspective of the application (though internally there is a fair amount of > flexibility on how to provide this guarantee). > I understand that processes that are detected (by the MPI lib) to fail will be stopped. That makes sense. What I do worry about is that a process or communication with a specific process is going awry. > It is the responsibility of the application to design around this type of > situation to ensure continued progress of their application - making a fault > aware application from an otherwise fault unaware application. There are a > few ways to do this, but the best solution will always be domain specific. > > OK > > It is possible that the application detects that a peer process is faulty > in a different way than fail-stop (e.g., byzantine). For example, a peer > process may have incurred a soft-error memory corruption, and is sending > invalid data (but valid from the MPI perspective). A peer process could be > checking the values, and determine that the peer is faulty. At which point > it can either: > - Coordinate with the other alive peers to exclude the faulty process, or > - Use MPI_Comm_kill() to request that the process be terminated. > MPI_Comm_kill() is not described in the user's guide, but is in the main > proposal. It allows one process to kill another without killing itself > (which would happen if they used MPI_Abort). Is this a scenario that you > were concerned about? > Yes, this is important to me. If not having a precise definition of what a 'failed' process actually is (and I understand why you avoid defining it), I'll design the app. such that it might itself decide that some (communication to a) process is failing and thus I want to be able to take the decision myself too. So this makes me feel more comfortable about it. > > > > Considering many apps work in master-slave mode, I would like to be able > to guarantee that the side on which the master resides is not killed. > > You should be able to do this by creating subsets of communicators and > setting error handers appropriately on the application end. Organize > 'workers' into volatile groups (error handler = MPI_ERRORS_ARE_FATAL), while > the 'manager' process(es) only ever participate with communicators that do > not have a fatal error handler. > > I don't think I understand. The master and slave need to have a common communicator which will be used to communicate between the two. If the associated error handler is MPI_ERRORS_RETURN and the communication between the two is cut, the MPI lib might decide to kill either of the two ? > Does that help clarify? > > A lot. Thanks! I think the way you (and others) clarified this, it would serve the user-guide too to discuss more in-depth what exactly a failure is, why it is not defined in detail and the ability for the app to detect errors and kill processes. toon

1 0

[Mpi3-ft] Review comments
by Moody, Adam T. 08 Feb '11

08 Feb '11

Hi Josh, Nice work putting all of this together. Here are some comments after reviewing the latest stabilization text. -Adam 1) In 2.4, define terms "unrecognized failure", "recognized failure" and "failed rank" up front. To new readers, later on it not's clear what "failed rank" means. 2) In 2.8, change "the application" to "a process" (semantics) and "terminates" to "terminated" (typo): "If the application receives notification of a process failure, the application can be assured that the specified process (identified by rank and generation) is terminates and is no longer participating in the job." One may misinterpret "the application" to mean "all processes in the application", which is incorrect. Maybe replace this with something like the following: "When a process receives notification that another process has failed, it may continue under the assumption that the failed process is no longer participating in the job. Even in cases where the failed process may not have terminated, the MPI implementation will ensure that no data is delivered to or from the failed process." 3) Examples of 3.10 3a) In the first example, there is no status object for MPI_Send. 3b) Some of the ANY_SOURCE examples are non-deterministic, e.g., some of the receive calls could return successfully because they also match the send from rank 3. In this case, the explicit receive for rank 3 will block and the application will not be notified of the rank 2 failure. 3c) To be again emphasize the local nature of process failure, it would be good to note before the examples that a failure denoted in the code represents the time at which the underlying MPI library of the calling process has been notified of the failure of specified rank. Or instead of "/*** Rank 2 failed ***/" change the comment to read "/*** MPI library of Rank 0 is notified that Rank 2 failed ***/" (or something like that). 4) Question about whether to leave receive buffer elements for failed processes to be "undefined" or "not modified". Allowing them to be undefined permits certain optimizations in MPI, however, it requires the application to keep a list of failed procs, whereas with not modified, this behaves more like PROC_NULL does today and the app can rely on MPI to keep track of the failed processes (by initializing the receive buffer with NULL data). 5) Question about whether to require all processes to continue calling all collectives even when a failure has occurred during an earlier collective. 6) In COMM_VALIDATE, want to set outcount to actual number needed in cases when incount is too small? Then app knows to call again with correct size for incount (unless there is another failure in between, in which case, the app can iterate again). Suggestion that the two variables could even be combined into a single INOUT. 7) In GROUP_VALIDATE, I take it that the process must list the ranks it wants to know about in the RANK_INFO list. However, do we have the user specify ranks via semi-opaque structures in other MPI functions, or should we just have the user specify ranks in a list of ints and make the RANK_INFO objects pure output? 8) Do we really want / need to provide group query functions, since users can not clear failures in a group anyway? 9) Want to require communicator creation calls to complete successfully everywhere? Can we instead just treat them as normal collectives by surronding them with comm_validate_all checks? 10) Checks for ERR_FAIL_STOP in irecv and isend calls in library examples, but I think early we say that errors will not be returned for non-blocking start calls. 11) MPI_ERRHANDLER_FREE -- error if freeing a predefined errhandler. At first, this seemed to be a good idea to me, but there is some inconsistency here, especially if you provide the user a way to compare errhandlers for equality. For example, what if the errhandler returned by GET_ERRHANDLER is a predefined errhandler? According to GET_ERRHANDLER, we always need to free the returned handle, but now we're saying we can't free predefined handlers. It seems ugly to require the user to check whether the handler is a predefined handler before calling ERRHANDLER_FREE. A similar problem: do we define what happens to MPI_COMM_FREE(MPI_COMM_WORLD)? 12) MPI_ERRHANDLER_COMPARE -- do we need this? 13) Many of the error classes seem to be a superset of what is needed in this proposal, e.g., MPI_ERR_CANNOT_CONTINUE, MPI_ERR_FN_CANNOT_CONTINUE, etc. I'd move these to text in a separate proposal. 14) When specifying a rank value in the context of an intercommunicator, that rank always refers to a process in the remote group. Thus, for an intercommunicator, COMM_VALIDATE_CLEAR cannot be used to clear ranks in the local group. Is there any problem there? Maybe not since you can't send pt-2-pt to processes in the local group? What about collectives? 15) I'd move the timeout discussion to a separate proposal page. It seems to be orthogonal. 16) For one-sided, why not handle put / gets to recognized failed procs like communication with PROC_NULL? And since we can't use MPI_COMM_VALIDATE_ALL on a window, how about something like a MPI_WIN_VALIDATE_ALL instead?

1 0

Re: [Mpi3-ft] Run-Through Stabilization Users Guide
by Joshua Hursey 08 Feb '11

08 Feb '11

On Feb 7, 2011, at 5:13 PM, Bronevetsky, Greg wrote: >> I think we should steer clear of even hinting to the application that we >> are going to kill processes so that we can save memory. Now the process >> could fail because MPI tried to allocate memory and ran out - at which >> point the MPI should return an error, and shutdown services - Maybe a good >> use case for CANNOT_CONTINUE. That situation is slightly different that >> proactively killing processes because we don't want to/cannot track their >> state. That sounds like a sure fire way to make users flee from the MPI to >> something else. >> > In that case, how do we deal with network partitions. As I mentioned in my email to Toon, the current semantics effectively force us to kill all the processes on one side of a network partition once the partition is healed. The same is true for any process that is erroneously judged to have failed because of a transient network problem. What do we do then? Note the following is one possible implementation solution that I have often seen in literature, there are likely others. For network partitions, the decision to terminate is (often) determined by a majority vote. So the group is the majority of alive processes wins (determined by the last known count of alive processes), and the group(s) with less than the majority self-terminate. If there is no clear majority (e.g., we divide into three even groups) then all groups will self terminate, and the application is terminated. So the application is never left in a split-brained scenario. The MPI implementation may choose (and should choose) to mask transient failures that it has already escalated to permanent failures. If the MPI implementation has not reported the transient process failure to the application when it receives notification of the process still being alive, then the MPI implementation needs to decide how to proceed. This is the scenario of building a perfect failure detector from an unreliable failure detector. It is a quality of implementation concern to how an MPI implementation handles transient failures that have not been reported to the application. If the MPI implementation has reported the failure to the application, then it must ensure that the process is failed-stop - either by killing it or masking its influence for the duration of the job. This must be guaranteed across the job, eventually - eventual consistency. There may be a period of time where one alive process sees a residual impression of a failed process due to network buffering. We cannot avoid this, but the MPI library should manage how this is relayed to the application depending on if it has decided this process to have failed yet or not. As far as how to terminate the processes in the minority partition (which may be what you were getting at), that is probably something we should clarify. An implementation may choose to shutdown the MPI library, effectively isolating the process. The MPI library would return CANNOT_CONTINUE for all following actions, and the application should cleanup and terminate as normal. Alternatively, the MPI implementation can forcibly terminate the processes in the minority group by calling MPI_Abort(COMM_SELF) in each of those processes. I think the first solution is a bit more kind to the application, and seems to fit with the wording of the current proposal. It provides the application with control over how it should shutdown, and save state if necessary. By extension, this scenario leads to how we determine what it means to "abort all tasks in the group of comm" for MPI_Abort(comm). We could either shutdown the MPI library and return CANNOT_CONTINUE (or maybe MPI_ERROR_YOU_HAVE_BEEN_ABORTED_BY_A_PEER), or have the MPI library forcibly terminate the other processes in the group (by calling MPI_Abort(COMM_SELF) if possible). I don't really have a strong leaning on the MPI_Abort() issue, and maybe we just leave it up to the implementation to define this scenario. Since if the MPI implementation forcibly terminates a process, that is effectively an unexpected fail-stop from the application perspective, and they are already setup to handle that. But on the flip side, giving the application more control, though isolated to operating by itself, is a bit more kind. The environment may dictate what is possible to provide in these scenarios. ... I donno, what do others think. > >> As far as the application is concerned, it only needs to allocate an array >> of MPI_Rank_info objects the size of that retuned by MPI_Comm_validate_all. >> Which should be small, but is inclusive of all the known failures >> (recognized and unrecognized). >> >> We could add a flag to the local MPI_Comm_validate that would return a list >> of the unrecognized failures, instead of all failures. Or, a bit more >> specifically, a 'state' key argument to the function, and it will return >> only those processes that match that state. This would allow the >> application to manage smaller sets of failures without having to iterate >> over previously recognized failures. What do you think about that? >> > I like both ideas very much. The use of a flag for MPI_Comm_validate can also be used MPI_Comm_validate_all as well and it avoids the need for two collectives. Also, having MPI return a list would be a big improvement. The set of failed processes will be sparse in the overall set of MPI ranks, so forcing developers to allocate and iterate over arrays will be very expensive. Lists solve this problem and in are a generally more scalable way to communicate information about a few ranks from the set of all possible ranks. We talked about this a bit more during the FT WG session at the MPI forum. We came up with a set of API solutions that we should discuss further. Since I was not able to hear the full conversation as accurately as I needed to, I asked that someone in the room post to the list the set of solutions for further discussion. But I think we are trending towards something along these lines. > >> Ah I see what you are saying now. So add one more state in order to the >> taxonomy: > ... >> Globally cleared really only has meaning at the full communicator level >> (used to determine if collectives are enabled/disabled). So wouldn't it be >> better to have a way to query the communicator to see if it was globally >> cleared or not, in order for the application to decide if collectives can >> be used or not. We talked a while back about adding an attribute to the >> communicator to indicate this, but I don't think it ever made it into the >> proposal. I'll make a note about this, and maybe we can discuss what it >> should look like next week. >> > Ah, now I see what You're saying! If any rank in a communicator is not globally cleared, then the only remedy is to clear the entire communicator. As such, the only useful API is one that checks the global-cleared status of communicators, not individual ranks. Yeah, that makes sense. Also, I think you should change the terminology from "globally" cleared to something more connected to communicators. For example, a rank may be "locally cleared" and a communicator can be "cleared" but a rank can't be "globally cleared" Getting a clear terminology for this is important as we move closer to standard language. So this is good. How about: An individual rank can be: - alive (MPI_RANK_STATE_OK) - unrecognized failed (MPI_RANK_STATE_FAILED) - locally cleared (MPI_RANK_STATE_NULL) A communicator can be (and by extension file handles and windows): - collectively cleared - collectively uncleared So we would use phrases like: Collectives: "the associated communicator must be collectively cleared before calling this function" P2P: "The associated rank must be alive or locally cleared before calling this function" I think that propagating the word 'collectively' with the state of the communicator (though redundant) helps clarify that clearing ranks locally does not lead to a clear communicator for collectives, the app. must call MPI_Comm_validate_all. Thanks for all the feedback, keep it coming :) -- Josh > > Greg Bronevetsky > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky(a)llnl.gov > http://greg.bronevetsky.com > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Bronevetsky, Greg 08 Feb '11

08 Feb '11

I'll add one more thing to Josh's explanation. A good analogy to the way we approached the FT spec is the design of the UI for a car. The sufficient UI is a steering wheel, a brake and a gearshift. But in practice users want to know more about the car to operate it efficiently: the maximum speed, the acceleration, the gearing ratios inside the transmission, the tread shape on the tires, etc. However, the latter properties are very implementation-specific, with different quality cars using different technologies to implement the car abstraction. In this case it is best to capture the minimum amount of information that is common to all implementations (wheel, brake, gearshift) and give implementers freedom to make either high-quality or low-quality implementations and independently inform the user about how well they perform. If you write an application using the MPI FT interface, you'll be guaranteed that your application will run on any system. If that system has a good implementation of this interface, failures will be detected quickly and accurately and it will thus be able to efficiently run on unreliable hardware. If the system has a poor implementation, your application will frequently waste cycles waiting on failed nodes that have not yet been detected or will abort mysteriously because of failures that were not at all detected by MPI. This MPI is a good fit for non-mission-critical applications or those running on reliable hardware. In either case, you can combine the MPI FT specification and the MPI implementer's specs to run your application efficiently on the hardware of your choice. Now, it would be nice to have a group outside of the MPI spec that looks at implementations and provides an unbiased guide about their performance and reliability. Unfortunately, such a guide is too low-level to include in the MPI specification. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Joshua Hursey > Sent: Tuesday, February 08, 2011 6:47 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] run through stabilization user-guide > > Toon, > > Thanks for your feedback. I'll fix the first example, thanks for the catch. > A bit more discussion on the other points below. > > On Feb 8, 2011, at 8:14 AM, Toon Knapen wrote: > > > > > The spec explicitly doesn't define the meaning of failure because it is a > very low-level concept. The MPI implementation is responsible for detecting > failures and defining what qualifies as one. The only guarantee that you > can rely on is that if the "failure" is bad enough that a given MPI rank > can't communicate with others, then MPI will have to either abort the > application completely or eventually report this failure via the FT API. > > > > > > I understand it is really hard to define what is exactly a failure. But > if the future standard does not describe it, is'nt there a risk that apps > that rely on the FT API will become hard to port from one implementation to > another ? > > > > For instance, suppose I have one process constantly sending messages to > the other one. Suppose the second process is dead, the first process thus > might either detect that the second process is down and inform the app. Or > the first process might not tell anything to the app and buffer all > messages to be send. But this buffer might grow so big that the first > process will run out-of-memory and dies too. > > > > So I think it would be usefull that the mpi-library is given some bounds > within which it should function and if it is not able to function within > these bounds it should raise an error. > > > > For instance a simple limit might be that each node should respond to a > ping within 1s (or 1ms. or ...). If a node can not be pinged (within this > limit), the node is considered to be dead. > > > > Another bound that might be specified and that would serve my example > above is e.g. the memory bounds within which MPI should function. For > instance, the app (or user) might decide to allocate 100 MB of (buffer-) > memory to MPI. If the app fills this buffer and the MPI-lib is not able to > flush the data sufficiently fast to the other nodes, an error will be > raised. At that point the app is aware that there is a failure to > communicate and can take appropriate action: slow down a bit with the > sending or consider the other node to have failed. > > > > In the above, it is the MPI lib that always detects the error. However if > the MPI library does not guarantee what a failure is exactly, I might have > to detect failures in the app (to be independent of the free interpretation > of failure). In that case I might e.g. need to always do non-blocking sends > and recvs (to avoid being blocked until eternity in case of a failure) and > see if the messages arrive in a timely manner, if not I consider the other > process to be dead. In that case however, I would also need functionality > to tell to the MPI-lib that a specific (comm,rank) should be considered > MPI_RANK_STATE_NULL. > > > > I'm sorry if the above has been discussed at length already and the forum > already decided not to define 'failure' (I should'nt have left the MPI- > scene for the last three years, what was I thinking ;-). Trying to define > 'failure' might be opening pandora's box but I'm looking at this from the > appication point-of-view. IMHO FT is either just about 'going down > gracefully' or about 'trying to finish the job'. > > It is important to clarify what we can and cannot specify in the standard. > So I appreciate your help with explaining how you think the language can be > more precise or better explained. > > For this proposal, we are focused on graceful degradation of the job. > Processes will fail(-stop), but the job as a whole is allowed the > opportunity to continue operating. > > We are defining process failures as fail-stop failures in the standard. So > we cover processes that crash, and do not continue to participate in the > parallel environment. How the MPI implementation detects process failure is > not defined by the standard, except by the properties that the MPI > implementation must provide to the application. The MPI implementation will > provide a view of the failure detector that is 'perfect' from the > perspective of the application (though internally there is a fair amount of > flexibility on how to provide this guarantee). > > This means that eventually all processes will know of a process failure, > and that if the application receives notification of process failure then > that process is guaranteed to be failed-stop. Once an alive process is > notified of a peer process failure then any communication involving the > failed process will complete with an error. So there is no worry that you > will block completely on communication to a failed peer. The MPI library > should be cleaning up internal buffers, etc at this time as well to > conserve memory. Communication with non-failed processes will complete as > normal. Collectives are disabled until the application re-enabled them with > the collective MPI_Comm_validate_all(). > > Now you could block in the following scenario: > - Processes A, B, D are alive, Process C is failed. > - Processes are communicating in a ring using point-to-point operations as > follows: A->B->C->D->A... > - Since processes B and D are directly interacting with C, then they will > see failure from their send/recv point-to-point communication. But A may > block waiting on either B or D. > > It is the responsibility of the application to design around this type of > situation to ensure continued progress of their application - making a > fault aware application from an otherwise fault unaware application. There > are a few ways to do this, but the best solution will always be domain > specific. > > > It is possible that the application detects that a peer process is faulty > in a different way than fail-stop (e.g., byzantine). For example, a peer > process may have incurred a soft-error memory corruption, and is sending > invalid data (but valid from the MPI perspective). A peer process could be > checking the values, and determine that the peer is faulty. At which point > it can either: > - Coordinate with the other alive peers to exclude the faulty process, or > - Use MPI_Comm_kill() to request that the process be terminated. > MPI_Comm_kill() is not described in the user's guide, but is in the main > proposal. It allows one process to kill another without killing itself > (which would happen if they used MPI_Abort). Is this a scenario that you > were concerned about? > > > As an aside, providing a memory boundary or specifying a heartbeat timeout > value is generally difficult to do in the standard. But I don't think you > necessarily need this functionality since it is a quality of implementation > issue on how soon the MPI library will notify the application of a process > failure, and how well it manages the buffers internally. > > > > > > > > Note that this view is focused on process failures. When it is applied to > things like network partitions (this includes the case you mentioned where > one process can't talk to any other due to a failed network card) then > processes on both parts of the partition may be informed that the others > have failed. As such, when connectivity is restored, since MPI will be > responsible to maintaining self-consistency of its previous notifications, > it'll have to kill processes on one side of the partition to keep > consistent with the notifications it gave to the other partition. > > > > > > Considering many apps work in master-slave mode, I would like to be able > to guarantee that the side on which the master resides is not killed. > > You should be able to do this by creating subsets of communicators and > setting error handers appropriately on the application end. Organize > 'workers' into volatile groups (error handler = MPI_ERRORS_ARE_FATAL), > while the 'manager' process(es) only ever participate with communicators > that do not have a fatal error handler. > > Does that help clarify? > > Thanks for the feedback, > Josh > > > > > > toon > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Graham, Richard L. 08 Feb '11

08 Feb '11

Quite a while back we decided that we were not going to handle byzantine failures. On 2/8/11 9:39 AM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: I believe the standard does define process failures as fail-stop. However, the standard does not describe how to detect failures. Doing this would severely limit implementations. I suspect that most implementations will rely on external mechanisms to detect process failure, such as ipmi, or a ras system. MPICH2 relies on the hydra process manager to detect abnormal process termination and node failures. I'm not sure whether fail-stop is exactly the same thing as abnormal process termination, but realistically this is how most vendors will implement it. Distinguishing between unresponsive processes and crashed processes is impossible to do. Even you add heartbeat messages between MPI libraries will only tell you that the library hasn't crashed. You could construct a scenario where the application is in an endless loop but doesn't call MPI_Recv, and the sender runs out of resources, even though the library is still alive. I believe crashed, but not terminated, would not be considered fail-stopped (but probably byzantine), and so outside the scope of the standard. Users that need to handle these kinds of failures would need to implement their own mechanism to detect hung processes, since they are in the best position to decide when a process is hung. -d On Feb 8, 2011, at 5:14 AM, Toon Knapen wrote: > > The spec explicitly doesn't define the meaning of failure because it is a very low-level concept. The MPI implementation is responsible for detecting failures and defining what qualifies as one. The only guarantee that you can rely on is that if the "failure" is bad enough that a given MPI rank can't communicate with others, then MPI will have to either abort the application completely or eventually report this failure via the FT API. > > > I understand it is really hard to define what is exactly a failure. But if the future standard does not describe it, is'nt there a risk that apps that rely on the FT API will become hard to port from one implementation to another ? > > For instance, suppose I have one process constantly sending messages to the other one. Suppose the second process is dead, the first process thus might either detect that the second process is down and inform the app. Or the first process might not tell anything to the app and buffer all messages to be send. But this buffer might grow so big that the first process will run out-of-memory and dies too. > > So I think it would be usefull that the mpi-library is given some bounds within which it should function and if it is not able to function within these bounds it should raise an error. > > For instance a simple limit might be that each node should respond to a ping within 1s (or 1ms. or ...). If a node can not be pinged (within this limit), the node is considered to be dead. > > Another bound that might be specified and that would serve my example above is e.g. the memory bounds within which MPI should function. For instance, the app (or user) might decide to allocate 100 MB of (buffer-) memory to MPI. If the app fills this buffer and the MPI-lib is not able to flush the data sufficiently fast to the other nodes, an error will be raised. At that point the app is aware that there is a failure to communicate and can take appropriate action: slow down a bit with the sending or consider the other node to have failed. > > In the above, it is the MPI lib that always detects the error. However if the MPI library does not guarantee what a failure is exactly, I might have to detect failures in the app (to be independent of the free interpretation of failure). In that case I might e.g. need to always do non-blocking sends and recvs (to avoid being blocked until eternity in case of a failure) and see if the messages arrive in a timely manner, if not I consider the other process to be dead. In that case however, I would also need functionality to tell to the MPI-lib that a specific (comm,rank) should be considered MPI_RANK_STATE_NULL. > > I'm sorry if the above has been discussed at length already and the forum already decided not to define 'failure' (I should'nt have left the MPI-scene for the last three years, what was I thinking ;-). Trying to define 'failure' might be opening pandora's box but I'm looking at this from the appication point-of-view. IMHO FT is either just about 'going down gracefully' or about 'trying to finish the job'. > > > Note that this view is focused on process failures. When it is applied to things like network partitions (this includes the case you mentioned where one process can't talk to any other due to a failed network card) then processes on both parts of the partition may be informed that the others have failed. As such, when connectivity is restored, since MPI will be responsible to maintaining self-consistency of its previous notifications, it'll have to kill processes on one side of the partition to keep consistent with the notifications it gave to the other partition. > > > Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed. > > toon > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] run through stabilization user-guide
by Joshua Hursey 08 Feb '11

08 Feb '11

Toon, Thanks for your feedback. I'll fix the first example, thanks for the catch. A bit more discussion on the other points below. On Feb 8, 2011, at 8:14 AM, Toon Knapen wrote: > > The spec explicitly doesnï¿½t define the meaning of failure because it is a very low-level concept. The MPI implementation is responsible for detecting failures and defining what qualifies as one. The only guarantee that you can rely on is that if the ï¿½failureï¿½ is bad enough that a given MPI rank canï¿½t communicate with others, then MPI will have to either abort the application completely or eventually report this failure via the FT API. > > > I understand it is really hard to define what is exactly a failure. But if the future standard does not describe it, is'nt there a risk that apps that rely on the FT API will become hard to port from one implementation to another ? > > For instance, suppose I have one process constantly sending messages to the other one. Suppose the second process is dead, the first process thus might either detect that the second process is down and inform the app. Or the first process might not tell anything to the app and buffer all messages to be send. But this buffer might grow so big that the first process will run out-of-memory and dies too. > > So I think it would be usefull that the mpi-library is given some bounds within which it should function and if it is not able to function within these bounds it should raise an error. > > For instance a simple limit might be that each node should respond to a ping within 1s (or 1ms. or ...). If a node can not be pinged (within this limit), the node is considered to be dead. > > Another bound that might be specified and that would serve my example above is e.g. the memory bounds within which MPI should function. For instance, the app (or user) might decide to allocate 100 MB of (buffer-) memory to MPI. If the app fills this buffer and the MPI-lib is not able to flush the data sufficiently fast to the other nodes, an error will be raised. At that point the app is aware that there is a failure to communicate and can take appropriate action: slow down a bit with the sending or consider the other node to have failed. > > In the above, it is the MPI lib that always detects the error. However if the MPI library does not guarantee what a failure is exactly, I might have to detect failures in the app (to be independent of the free interpretation of failure). In that case I might e.g. need to always do non-blocking sends and recvs (to avoid being blocked until eternity in case of a failure) and see if the messages arrive in a timely manner, if not I consider the other process to be dead. In that case however, I would also need functionality to tell to the MPI-lib that a specific (comm,rank) should be considered MPI_RANK_STATE_NULL. > > I'm sorry if the above has been discussed at length already and the forum already decided not to define 'failure' (I should'nt have left the MPI-scene for the last three years, what was I thinking ;-). Trying to define 'failure' might be opening pandora's box but I'm looking at this from the appication point-of-view. IMHO FT is either just about 'going down gracefully' or about 'trying to finish the job'. It is important to clarify what we can and cannot specify in the standard. So I appreciate your help with explaining how you think the language can be more precise or better explained. For this proposal, we are focused on graceful degradation of the job. Processes will fail(-stop), but the job as a whole is allowed the opportunity to continue operating. We are defining process failures as fail-stop failures in the standard. So we cover processes that crash, and do not continue to participate in the parallel environment. How the MPI implementation detects process failure is not defined by the standard, except by the properties that the MPI implementation must provide to the application. The MPI implementation will provide a view of the failure detector that is 'perfect' from the perspective of the application (though internally there is a fair amount of flexibility on how to provide this guarantee). This means that eventually all processes will know of a process failure, and that if the application receives notification of process failure then that process is guaranteed to be failed-stop. Once an alive process is notified of a peer process failure then any communication involving the failed process will complete with an error. So there is no worry that you will block completely on communication to a failed peer. The MPI library should be cleaning up internal buffers, etc at this time as well to conserve memory. Communication with non-failed processes will complete as normal. Collectives are disabled until the application re-enabled them with the collective MPI_Comm_validate_all(). Now you could block in the following scenario: - Processes A, B, D are alive, Process C is failed. - Processes are communicating in a ring using point-to-point operations as follows: A->B->C->D->A... - Since processes B and D are directly interacting with C, then they will see failure from their send/recv point-to-point communication. But A may block waiting on either B or D. It is the responsibility of the application to design around this type of situation to ensure continued progress of their application - making a fault aware application from an otherwise fault unaware application. There are a few ways to do this, but the best solution will always be domain specific. It is possible that the application detects that a peer process is faulty in a different way than fail-stop (e.g., byzantine). For example, a peer process may have incurred a soft-error memory corruption, and is sending invalid data (but valid from the MPI perspective). A peer process could be checking the values, and determine that the peer is faulty. At which point it can either: - Coordinate with the other alive peers to exclude the faulty process, or - Use MPI_Comm_kill() to request that the process be terminated. MPI_Comm_kill() is not described in the user's guide, but is in the main proposal. It allows one process to kill another without killing itself (which would happen if they used MPI_Abort). Is this a scenario that you were concerned about? As an aside, providing a memory boundary or specifying a heartbeat timeout value is generally difficult to do in the standard. But I don't think you necessarily need this functionality since it is a quality of implementation issue on how soon the MPI library will notify the application of a process failure, and how well it manages the buffers internally. > > > Note that this view is focused on process failures. When it is applied to things like network partitions (this includes the case you mentioned where one process canï¿½t talk to any other due to a failed network card) then processes on both parts of the partition may be informed that the others have failed. As such, when connectivity is restored, since MPI will be responsible to maintaining self-consistency of its previous notifications, itï¿½ll have to kill processes on one side of the partition to keep consistent with the notifications it gave to the other partition. > > > Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed. You should be able to do this by creating subsets of communicators and setting error handers appropriately on the application end. Organize 'workers' into volatile groups (error handler = MPI_ERRORS_ARE_FATAL), while the 'manager' process(es) only ever participate with communicators that do not have a fatal error handler. Does that help clarify? Thanks for the feedback, Josh > > toon > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0