June 2011 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] New draft of the Run-Through Stabilization document
by Josh Hursey 15 Jun '11

15 Jun '11

I just uploaded a new draft of the RTS proposal to the wiki. https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… Changes: * 2.8: Removed "Similarly, MPI itself provides no mechanisms for handling processor failures." and added the paragraph/advice discussed on the list. * 8.3: Removed modification on the end of the paragraph starting with "After an error is detected," * 13.7: Removed previous update to the Advice to Users based upon clarification in chapter 2. * 17.1: Adjusted last sentence of Advice to Implementors per teleconf. * 17.2: Added definition/citation for 'error' and 'failure' * 17.3: Fixed plural typo in Rationale. * 17.4: Took a pass at clarifying that the process state attribute is consistent to the local process, not all ranks in the application. * 17.5: Fixed a bunch of "rank_states" that should have been "rank_infos" in the function argument declarations. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] FT WG Teleconf Today @ Noon EDT
by Josh Hursey 15 Jun '11

15 Jun '11

On the call today we decided to meet again on Monday, June 20 at 11:30 am EDT/New York for an hour to continue working through the RTS proposal. I'll send out a reminder the morning of. We will also meet on Wednesday, June 22 at the normal time to continue the discussion. Thanks, Josh On Wed, Jun 15, 2011 at 9:00 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Reminder that we have a teleconf today, Wed 6/15/2011, at 12 noon EDT. > > Date: June 15, 2011 > Time: 12 noon EDT/New York > Dial-in information: > 877-801-8130 > Code: 1044056 > > Agenda: > - Run-through stabilization proposal review > ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > ï¿½- https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… > > Next Meetings: > * June ï¿½15, 2011 (Teleconf) > * June ï¿½22, 2011 (Teleconf) > * June ï¿½29, 2011 (Teleconf) > * July ï¿½ 6, 2011 (Teleconf) > * July ï¿½18, 2011 (MPI Forum Meeting) > * Aug. ï¿½ 3, 2011 (Teleconf) > * Aug. ï¿½17, 2011 (Teleconf) > * Aug. ï¿½31, 2011 (Teleconf) > * Sept. 14, 2011 (Teleconf) > * Sept. 22, 2011 (MPI Forum Meeting) > > Respond to the list if you have any additional topics that you would > like to discuss. > > -- Josh > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] The state of MPI is undefined
by Josh Hursey 15 Jun '11

15 Jun '11

On Mon, Jun 13, 2011 at 5:28 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > On Jun 13, 2011, at 3:36 PM, Josh Hursey wrote: > >> On Mon, Jun 13, 2011 at 12:36 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >>> >>> I'm not sure I like the wording: >>> >>> ï¿½If the MPI implementation can continue operating after process failure >>> ï¿½then it must return an appropriate error class (e.g., >>> ï¿½MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined >>> ï¿½in Chapter 17. >>> >>> I don't think we can say that "if MPI returns error code X" ==> "MPI state is defined," because if some state corrupting error occurs, the state of MPI is undefined and the implementation might return error code X. ï¿½Rather, I think we should use the form "if error FOO occurs" ==> "MPI state will be BAR." ï¿½So for example, we can say "if an unrecoverable communication error occurs, the MPI implementation will return a MPI_ERR_COMMUNICATION for all pending communication operations and blah, blah, blah." >> >> For the case where the MPI was corrupted so that it is forced to >> return a particular error code, I do not know how we can get around >> the fact that we cannot define the state of MPI (the error forced the >> MPI implementation to lie to the application). So even in the form "if >> error FOO occurs" ==> "MPI state will be BAR" the 'will be' is too >> strong since the corruption forced the return of FOO without the >> consent of the MPI implementation. ï¿½But if we say 'might be' then we >> are back to introducing a proposal that cannot be used by applications >> - lack of solid semantics. > > Sorry, I should have been more specific. ï¿½When I say "if error FOO occurs" ï¿½I don't mean error FOO is returned but rather that an error condition occurred. ï¿½E.g., if a communication error occurs, an MPI_ERR_COMMUNICATION error will be returned." ï¿½So this explicitly excludes the state corrupting error. ï¿½Note that it's still possible that a state corrupting error results in MPI_ERR_COMMUNICATION to be returned, but we can't prevent that case. ï¿½But at least we can say that. e.g., assuming the state of the library is not already corrupted, for a compliant implementation, a communication error (however we define that) will not corrupt the library and the library will behave as prescribed. Ah I see what you mean now. I think we should be aware of the corner case and point it out to the users, but try not to have it diminish the strength of the language of the standard as much as possible. Per the advice to users below. > >> >> So maybe we should add an advice to users that states that: >> ------------ >> It is possible that the state of MPI becomes corrupted in an >> undetectable manner. This undetectable error might force the MPI >> implementation to return an error code (or even success) >> unintentionally. For this pathological case there exist no mechanisms >> to determine the correct running state. A high-quality implementation >> will strive to return the correct return code from all MPI operations. >> ------------ > > I think this is OK, but maybe we can make it more general: > > It is possible that some error/fault* may cause the state of MPI to become corrupted in an undetectable manner. ï¿½The behavior of MPI in this case is undefined, and it is possible that the implementation returns incorrect error codes (or even MPI_SUCCESS). ï¿½So while a high-quality implementation will strive to always return correct return codes from MPI operations, it may not be possible in all cases. I think that rewording of the advice to users is fine. Below is slightly touched up: -------------- It is possible that some error may cause the state of MPI to become corrupted in an undetectable manner. The behavior of MPI in this case is undefined, and it is possible that the implementation returns incorrect error codes (including MPI_SUCCESS). So while a high-quality implementation will strive to always return correct return codes from MPI operations, it may not be possible in all cases. -------------- > > > * Perhaps we should define error and fault and check for correct usage in the standard. Does anyone have good definitions? ï¿½Is an error a detected (at some level) fault? ï¿½Should we drop error and just use fault? I have used the following before (there may be other, better definitions out there): --------------------- In a single system, an error is generated when a physical defect, called a fault, is detected. A system failure is when the system cannot deliver its intended function because of one or more errors [A]. A fault-tolerant system will continue to operate normally in the presence of errors. [A] http://portal.acm.org/citation.cfm?id=1207694 --------------------- -- Josh > > -d > >> >>> >>> A blanket statement we can make about errors, then, would be something like "Unless otherwise noted in the standard, after an error is detected, the state of MPI is undefined." >>> >>> Another thought, we may get resistance from the Forum about the cost of checking for faults. ï¿½Maybe we discussed this already, I can't remember. ï¿½Do we want to make FT optional, and say "If the MPI implementation supports FT, then the behavior will be blah"? >> >> How about: >> ------------------ >> MPI does not provide the user with transparent process recovery upon >> process failure. >> Once a process fails, MPI does not guarantee that the job can continue >> or, if the job can continue, that the process can be recovered. >> If a process failure occurs, and the MPI implementation is able to >> correctly continue operating after process failure then the MPI >> implementation will return an appropriate error class (e.g., >> MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined >> in Chapter 17. >> The MPI implementation documentation will provide information on the >> possible effect of each supported class of errors. >> >> Advice to users: >> It is possible that the state of MPI becomes corrupted in an >> undetectable manner. This undetectable error might force the MPI >> implementation to return an error code (or even success) >> unintentionally. For this pathological case there exist no mechanisms >> to determine the correct running state. A high-quality implementation >> will strive to return the correct return code from all MPI operations. >> ------------------ >> >> I like the idea of keeping the softer language with regard to whether >> or not it is required, but instead say that if the user sees error >> code X (in this case MPI_ERR_RANK_FAIL_STOP) then they can expect the >> behavior outlined in Chapter 17. If they do not see that error, but >> something else then it is implementation defined what will happen >> afterwards. And the corner case is described to the users, which is an >> understandable limitation of any software system. >> >> >> What do folks think about this? >> >> -- Josh >> >> >>> >>> -d >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] General Network Channel Failure
by Josh Hursey 15 Jun '11

15 Jun '11

I agree that middleware will have to step up and help support an application in handling failures. Further, the MPI specification for handling faults should fully support middleware/libraries that want to do this - in fact that is part of the mission of this group to define the minimal interface required to do so. My hesitation about a query interface is that it seems like it trends towards not specifying anything in the MPI standard about expected behavior after a failure - which could make it difficult/impossible to use. If we only specify that a query interface for the programmer to use to ask for what the MPI implementation can provide at the moment after a failure, and then expect the application/middleware to adjust dynamically to the set of operations available then this becomes difficult to program - even for middleware. The space of possible responses from the implementation may be to large to enumerate. A query interface would have to categorize all possible responses an MPI could give to its current capabilities (some may not be known until the operation is called, or change over time). For example, broadcast works after error X as long as it is from ranks 0-5 with datatype X and count no greater than Y. Or operation MPI_Foo and MPI_Bar work but may not be used together in the same epoch. The MPI_ERR_CANNOT_CONTINUE error class (currently dropped from the RTS proposal) was a gesture towards the idea of partially supporting the defined semantics. So if an MPI implementation can only provide some but not all of the functionality required by the standard after a particular failure, then it would return some variation of MPI_ERR_CANNOT_CONTINUE for that operation - or that operation with the specified arguments. It was not ideal, and maybe the query interface in a natural evolution of a solution to a similar problem. I guess I need some more specifics about what a query interface would look like for a certain class of errors to better understand the use cases. Do you want to start a wiki page with some details and examples? -- Josh On Tue, Jun 14, 2011 at 1:10 PM, Kathryn Mohror <kathryn(a)llnl.gov> wrote: > Hi Josh, all, > > On 6/13/2011 5:06 PM, Bronevetsky, Greg wrote: >> >> I like the idea of a query interface. This is primarily because I >> believe that fault tolerance should be the responsibility of >> middleware rather than applications. Writing scalable codes is >> already hard and writing your own reliability solutions that are also >> scalable and robust is almost certainly too much to ask for. However, >> if we leave the job to middleware, we'll need to provide an interface >> that is both portable and flexible. > > Yes, this is the idea I was going for -- that fault tolerance would be best > handled by libraries/middleware rather than by applications directly. Then, > applications don't need to focus on the complexities of dealing with a query > interface for MPI implementation specific error handling. That said, I am > not sure if this interface would belong in the MPI_T interface, since that > was written with gathering/controlling performance in mind. However, if the > intent is to have the interface be used primarily by tools (and rogue power > application writers), then maybe the tools chapter is the best place for a > query/control interface for fault tolerance, possibly under a different name > space, MPI_FT?. > > Kathryn > >> >> If we go for an MPI-level interface we get portability but we also >> get complexity because MPI provides developers with a large number of >> capabilities (send to/receive from a rank, collectively communicate >> on communicator, read/write a file, etc.). A full API will need to >> allow the application to perform queries to identify which >> capabilities are available after a given fault. A decent API for this >> might be to allow users to query for individual capabilities and also >> allow for short-cut APIs. For example, the run-through API focuses on >> fail-stop failures of one or more ranks. As such, if MPI reports that >> a rank has failed, this implies that a fixed set of capabilities is >> no longer available. We can extend this idea to other failure models >> that virtualize sets of capabilities under a single name, meaning >> that a problem with a single virtual capability (slow network or >> corrupted file) implies a specific change to large number of >> capabilities (all sends/receives are slow or read! s/writes to just >> this file are invalid). >> >> Further, failure models themselves do not need to be specified in the >> spec. The reason this might make sense is if we want to model various >> levels using different abstractions. A simple way to represent >> network failures is to treat the network as a monolithic entity. In >> this model MPI_NETWORK_ERROR means that no communication can take >> place. A more precise abstraction might treat the network as a set of >> communication islands and one island can fail without causing errors >> for communication on other islands. An even more precise model may >> more individual rank-to-rank connections ï¿½and allow the user to >> identify which connections are valid/invalid after a failure. All >> these models are just different portable approximations of the real >> system state and are useful for different applications. Right now I'm >> nervous about choosing one to be in the spec while leaving the others >> out. However, it should be pretty useful to force MPI to implement >> some fairly detailed choice and leave the! higher-level ones to be >> implemented outside the spec but provide a standard interface to >> present them to users. >> >> Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 >> bronevetsky(a)llnl.gov http://greg.bronevetsky.com >> >> >>> -----Original Message----- From: >>> mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey Sent: Monday, >>> June 13, 2011 2:06 PM To: MPI 3.0 Fault Tolerance and Dynamic >>> Process Control working Group Subject: Re: [Mpi3-ft] General >>> Network Channel Failure >>> >>> Yeah the 'system model' is probably not going to fly with the MPI >>> Forum. Mostly because we would be in the business then of defining >>> what a 'node', 'NIC', 'RAM', etc. are - something the standard >>> strives to not defined for future proofing. >>> >>> So I like the MPI model a bit more. For resource exhaustion we do >>> get a bit of wiggle room with the 'reliable communication' >>> requirement in 3.7 "If the call causes some system resource to be >>> exhausted, then it will fail and return an error code." And for >>> memory in 8.2 "The function MPI_ALLOC_MEM may return an error code >>> of class MPI_ERR_NO_MEM to indicate it failed because memory is >>> exhausted." >>> >>> So even though MPI shys away from defining network resources and >>> memory, we might be able to find some acceptable language within >>> the confines of these few error statements. So I think we can talk >>> about connections between pairs (P2P) and groups (collectives) of >>> processes, and maybe even connections to files (I/O). For each >>> error class that we want to define we need to go through the >>> process of asking "so if this error code was returned from this >>> operation, what would that mean to the programmer? How do we want >>> them to react to it?" Taking resource exhaustion as an example, for >>> point-to-point messages if the application receives such an error >>> then they may want to repost the message later. But if the same >>> error is returned locally from a collective operation we need to >>> determine if the user should repost the same collective or if the >>> collective is guaranteed to return at all other processes with >>> either success or some other error. >>> >>> I believe that the primitives that we specified for fail-stop >>> process failure can likely be extended to support other classes of >>> errors without too much modification. But we will have to see as we >>> go along how true that becomes. >>> >>> >>> For Greg, this is the thread of conversation in which we will >>> likely want to explore transitent failures and possibly even revive >>> the discussion about performance degradation notification (probably >>> performance notification is best saved for another thread). But >>> thinking about modern failure detectors that provide an assurance >>> range (accural failure detectors are what I am thinking of >>> specifically) then the MPI may want to return an error indicating >>> something symptomatic of the eventual diagnosis of a process as >>> either fail-stop or transient. >>> >>> >>> To Kathryn's point about the MPI_T proposal, so you are suggesting >>> that we provide a query interface that the application can use to >>> determine what types of errors can be handled and how the MPI >>> implementation allows the application to continue after them? I >>> think this is a good/necessary idea for tools support - so maybe as >>> an extension to the MPI_T proposal. But I am hesitant to do so as >>> the main mechanism for processing emerging errors. I think that >>> applications would struggle with something that flexible since, it >>> seems to imply that, they would have to adapt not to the error >>> code, but to the error code and reported capabilities of the MPI >>> implementation. So for the programmer they would need to have a >>> check for the error code, a query for capabilities, and then a >>> switch statement for all the possible actions that they can take. I >>> like the flexibility, but I think it becomes too much of a burden >>> on the programmer. >>> >>> I guess the counter argument would be that the MPI implementations >>> should be, today, documenting all expected errors that they could >>> return to the user. The MPI_T-like interface just provides a >>> programatic way to access this information and react to it >>> dynamically, instead of relying on documentation updates and a >>> round of software updates. So this would fit nicely into the >>> existing requirements for MPI implementations, though at some >>> additional complexity to the end user. >>> >>> Interesting idea, if I understand it correctly. What do others >>> think? >>> >>> >>> -- Josh >>> >>> >>> On Sun, Jun 12, 2011 at 12:09 PM, Kathryn Mohror<kathryn(a)llnl.gov> >>> wrote: >>>> >>>> Hi all, >>>> >>>> (I sent this earlier, but I don't think it went through because I >>>> sent it from the wrong email address. I apologize if you get an >>>> extra copy.) >>>> >>>> I know I haven't participated in this working group yet, so I may >>>> be missing some context, but I couldn't resist putting my two >>>> cents in! >>>> >>>> I think that an MPI-centric approach is best. Otherwise, you run >>>> the risk of defining a model that doesn't fit with a particular >>>> implementation or machine and get shot down when it's brought to >>>> the forum. For example, you may remember the PERUSE performance >>> >>> interface >>>> >>>> that assumed a model of MPI that implementers didn't approve, >>>> because it didn't fit their implementation or was >>>> difficult/expensive to support. Now, to replace PERUSE, we've got >>>> the MPI_T interface which doesn't specify *anything* but appears >>>> to be supported by the forum. >>>> >>>> I agree though that having more specific error information when >>>> it's available would be very useful. You might consider taking an >>>> approach similar to MPI_T -- allow MPI implementers to define any >>>> specific error codes they can/want and then provide an interface >>>> for decoding and interpreting the errors. >>>> >>>> Of course, this approach may not be useful for most applications >>>> directly, but I imagine that a fault-tolerant MPI application or >>>> a checkpoint/restart library could make use of the information, >>>> assuming they could get at it. >>>> >>>> Kathryn >>>> >>>> >>>> >>>> >>>> On 6/9/2011 8:20 AM, Howard Pritchard wrote: >>>>> >>>>> Hi Greg, >>>>> >>>>> I vote for an MPI-centric model. >>>>> >>>>> I also think that part of the job of MPI is to hide as much as >>>>> possible things like 'exhaustion of network resources' and >>>>> 'intermittent network failures'. ï¿½Indeed, the very first >>>>> sentence in section 2.8 says "MPI provides the user with >>>>> reliable message transmission". >>>>> >>>>> The only reason the topic came up yesterday was in the context >>>>> of the fail-stop model and what types of error codes might be >>>>> returned by MPI before the official verdict was in that a >>>>> fail-stop had occurred. Several of us checked what our >>>>> implementations might do prior to that, and it could include >>>>> returning MPI_ERR_OTHER. ï¿½I could see how for someone writing a >>>>> fault tolerant MPI application, something more useful than this >>>>> rather ambiguous error code might be worth defining. >>>>> >>>>> Howard >>>>> >>>>> >>>>> Bronevetsky, Greg wrote: >>>>>> >>>>>> I like the idea of having an abstract model of failures that >>>>>> can approximate changes in system functionality due to >>>>>> failures. However, I think before we go too far with this we >>>>>> should consider the type of model we want to make. One option >>>>>> is to make a system model that has as its basic elements >>>>>> nodes, network links and other hardware components and >>>>>> identifies points in time when stop functioning. The other >>>>>> option is to make it MPI-centric by talking about the status >>>>>> of ranks and point-to-point communication between them as >>>>>> well as communicators and collective communication over them. >>>>>> So in the first type of model we can talk about network >>>>>> resource exhaustion and in the latter we can talk about an >>>>>> intermittent >>> >>> inability to send messages over some or all communicators. >>>>>> >>>>>> I think that the MPI-centric model is a better option since >>>>>> it talks exclusively ï¿½about entities that exist in MPI and >>>>>> ignores the physical phenomena that cause a given type of >>>>>> degradation in >>> >>> functionality. >>>>>> >>>>>> The other question we need to discuss is the types of >>>>>> problems we want to represent. We obviously care about >>>>>> fail-stop failures but we're not talking about resource >>>>>> exhaustion. Do we want to add error classes for transient >>>>>> errors and if so, what about performance >>> >>> slowdowns? >>>>>> >>>>>> Greg Bronevetsky Lawrence Livermore National Lab (925) >>>>>> 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com >>>>>> >>>>>> >>>>>>> -----Original Message----- From: >>>>>>> mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey Sent: >>>>>>> Wednesday, June 08, 2011 11:36 AM To: MPI 3.0 Fault >>>>>>> Tolerance and Dynamic Process Control working Group >>>>>>> Subject: [Mpi3-ft] General Network Channel Failure >>>>>>> >>>>>>> It was mentioned in the conversation today that >>>>>>> MPI_ERR_RANK_FAIL_STOP may not be the first error returned >>>>>>> by an MPI call. In particular the MPI call may return an >>>>>>> error symptomatic of a fail-stop process failure (e.g., >>>>>>> network link failed - currently MPI_ERR_OTHER), before >>>>>>> eventually diagnosing the event as a process failure. This >>>>>>> 'space between' MPI_SUCCESS behavior and >>>>>>> MPI_ERR_RANK_FAIL_STOP behavior is not currently defined, >>>>>>> and probably should be for the application to cleanly move >>>>>>> from set of semantics for one error class to another. >>>>>>> >>>>>>> The suggestion was to create a new general network error >>>>>>> class (e.g., MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - >>> >>> MPI_ERR_COMM is >>>>>>> >>>>>>> taken) that can be returned when the operation cannot >>>>>>> complete due to network issues (which might be later >>>>>>> diagnosed as process failure and escalated to the >>>>>>> MPI_ERR_RANK_FAIL_STOP semantics). >>> >>> You >>>>>>> >>>>>>> could also think about this error being used for network >>>>>>> resource exhaustion as well (something that Tony mentioned >>>>>>> during the last MPI Forum meeting). In which case retrying >>>>>>> at a later time or taking some other action before trying >>>>>>> again would be useful/expected. >>>>>>> >>>>>>> There are some issues with matching, and the implications >>>>>>> on collective operations. If the network error is >>>>>>> sticky/permanent then once the error is returned it will >>>>>>> always be returned or escalated to fail-stop process >>>>>>> failure (or more generally to a 'higher/more severe/more >>>>>>> detailed' error class). A recovery proposal (similar to >>>>>>> what we are developing for process failure) would allow the >>>>>>> application to 'recover' the channel and continue >>>>>>> communicating on it. >>>>>>> >>>>>>> >>>>>>> The feeling was that this should be expanded into a full >>>>>>> proposal, separate from the Run-Through Stabilization >>>>>>> proposal. So we can continue with the RTS proposal, and >>>>>>> bring this forward when it is ready. >>>>>>> >>>>>>> >>>>>>> What to folks think about this idea? >>>>>>> >>>>>>> -- Josh >>>>>>> >>>>>>> -- Joshua Hursey Postdoctoral Research Associate Oak Ridge >>>>>>> National Laboratory http://users.nccs.gov/~jjhursey >>>>>>> _______________________________________________ mpi3-ft >>>>>>> mailing list mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> _______________________________________________ mpi3-ft >>>>>> mailing list mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>> _______________________________________________ mpi3-ft mailing >>>> list mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>> >>> >>> >>> -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National >>> Laboratory http://users.nccs.gov/~jjhursey >>> >>> _______________________________________________ mpi3-ft mailing >>> list mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT WG Teleconf Today @ Noon EDT
by Josh Hursey 15 Jun '11

15 Jun '11

Reminder that we have a teleconf today, Wed 6/15/2011, at 12 noon EDT. Date: June 15, 2011 Time: 12 noon EDT/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Run-through stabilization proposal review - https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/run_through… Next Meetings: * June 15, 2011 (Teleconf) * June 22, 2011 (Teleconf) * June 29, 2011 (Teleconf) * July 6, 2011 (Teleconf) * July 18, 2011 (MPI Forum Meeting) * Aug. 3, 2011 (Teleconf) * Aug. 17, 2011 (Teleconf) * Aug. 31, 2011 (Teleconf) * Sept. 14, 2011 (Teleconf) * Sept. 22, 2011 (MPI Forum Meeting) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] General Network Channel Failure
by Kathryn Mohror 14 Jun '11

14 Jun '11

Hi Josh, all, On 6/13/2011 5:06 PM, Bronevetsky, Greg wrote: > I like the idea of a query interface. This is primarily because I > believe that fault tolerance should be the responsibility of > middleware rather than applications. Writing scalable codes is > already hard and writing your own reliability solutions that are also > scalable and robust is almost certainly too much to ask for. However, > if we leave the job to middleware, we'll need to provide an interface > that is both portable and flexible. Yes, this is the idea I was going for -- that fault tolerance would be best handled by libraries/middleware rather than by applications directly. Then, applications don't need to focus on the complexities of dealing with a query interface for MPI implementation specific error handling. That said, I am not sure if this interface would belong in the MPI_T interface, since that was written with gathering/controlling performance in mind. However, if the intent is to have the interface be used primarily by tools (and rogue power application writers), then maybe the tools chapter is the best place for a query/control interface for fault tolerance, possibly under a different name space, MPI_FT?. Kathryn > > If we go for an MPI-level interface we get portability but we also > get complexity because MPI provides developers with a large number of > capabilities (send to/receive from a rank, collectively communicate > on communicator, read/write a file, etc.). A full API will need to > allow the application to perform queries to identify which > capabilities are available after a given fault. A decent API for this > might be to allow users to query for individual capabilities and also > allow for short-cut APIs. For example, the run-through API focuses on > fail-stop failures of one or more ranks. As such, if MPI reports that > a rank has failed, this implies that a fixed set of capabilities is > no longer available. We can extend this idea to other failure models > that virtualize sets of capabilities under a single name, meaning > that a problem with a single virtual capability (slow network or > corrupted file) implies a specific change to large number of > capabilities (all sends/receives are slow or read! s/writes to just > this file are invalid). > > Further, failure models themselves do not need to be specified in the > spec. The reason this might make sense is if we want to model various > levels using different abstractions. A simple way to represent > network failures is to treat the network as a monolithic entity. In > this model MPI_NETWORK_ERROR means that no communication can take > place. A more precise abstraction might treat the network as a set of > communication islands and one island can fail without causing errors > for communication on other islands. An even more precise model may > more individual rank-to-rank connections and allow the user to > identify which connections are valid/invalid after a failure. All > these models are just different portable approximations of the real > system state and are useful for different applications. Right now I'm > nervous about choosing one to be in the spec while leaving the others > out. However, it should be pretty useful to force MPI to implement > some fairly detailed choice and leave the! higher-level ones to be > implemented outside the spec but provide a standard interface to > present them to users. > > Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 > bronevetsky(a)llnl.gov http://greg.bronevetsky.com > > >> -----Original Message----- From: >> mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey Sent: Monday, >> June 13, 2011 2:06 PM To: MPI 3.0 Fault Tolerance and Dynamic >> Process Control working Group Subject: Re: [Mpi3-ft] General >> Network Channel Failure >> >> Yeah the 'system model' is probably not going to fly with the MPI >> Forum. Mostly because we would be in the business then of defining >> what a 'node', 'NIC', 'RAM', etc. are - something the standard >> strives to not defined for future proofing. >> >> So I like the MPI model a bit more. For resource exhaustion we do >> get a bit of wiggle room with the 'reliable communication' >> requirement in 3.7 "If the call causes some system resource to be >> exhausted, then it will fail and return an error code." And for >> memory in 8.2 "The function MPI_ALLOC_MEM may return an error code >> of class MPI_ERR_NO_MEM to indicate it failed because memory is >> exhausted." >> >> So even though MPI shys away from defining network resources and >> memory, we might be able to find some acceptable language within >> the confines of these few error statements. So I think we can talk >> about connections between pairs (P2P) and groups (collectives) of >> processes, and maybe even connections to files (I/O). For each >> error class that we want to define we need to go through the >> process of asking "so if this error code was returned from this >> operation, what would that mean to the programmer? How do we want >> them to react to it?" Taking resource exhaustion as an example, for >> point-to-point messages if the application receives such an error >> then they may want to repost the message later. But if the same >> error is returned locally from a collective operation we need to >> determine if the user should repost the same collective or if the >> collective is guaranteed to return at all other processes with >> either success or some other error. >> >> I believe that the primitives that we specified for fail-stop >> process failure can likely be extended to support other classes of >> errors without too much modification. But we will have to see as we >> go along how true that becomes. >> >> >> For Greg, this is the thread of conversation in which we will >> likely want to explore transitent failures and possibly even revive >> the discussion about performance degradation notification (probably >> performance notification is best saved for another thread). But >> thinking about modern failure detectors that provide an assurance >> range (accural failure detectors are what I am thinking of >> specifically) then the MPI may want to return an error indicating >> something symptomatic of the eventual diagnosis of a process as >> either fail-stop or transient. >> >> >> To Kathryn's point about the MPI_T proposal, so you are suggesting >> that we provide a query interface that the application can use to >> determine what types of errors can be handled and how the MPI >> implementation allows the application to continue after them? I >> think this is a good/necessary idea for tools support - so maybe as >> an extension to the MPI_T proposal. But I am hesitant to do so as >> the main mechanism for processing emerging errors. I think that >> applications would struggle with something that flexible since, it >> seems to imply that, they would have to adapt not to the error >> code, but to the error code and reported capabilities of the MPI >> implementation. So for the programmer they would need to have a >> check for the error code, a query for capabilities, and then a >> switch statement for all the possible actions that they can take. I >> like the flexibility, but I think it becomes too much of a burden >> on the programmer. >> >> I guess the counter argument would be that the MPI implementations >> should be, today, documenting all expected errors that they could >> return to the user. The MPI_T-like interface just provides a >> programatic way to access this information and react to it >> dynamically, instead of relying on documentation updates and a >> round of software updates. So this would fit nicely into the >> existing requirements for MPI implementations, though at some >> additional complexity to the end user. >> >> Interesting idea, if I understand it correctly. What do others >> think? >> >> >> -- Josh >> >> >> On Sun, Jun 12, 2011 at 12:09 PM, Kathryn Mohror<kathryn(a)llnl.gov> >> wrote: >>> Hi all, >>> >>> (I sent this earlier, but I don't think it went through because I >>> sent it from the wrong email address. I apologize if you get an >>> extra copy.) >>> >>> I know I haven't participated in this working group yet, so I may >>> be missing some context, but I couldn't resist putting my two >>> cents in! >>> >>> I think that an MPI-centric approach is best. Otherwise, you run >>> the risk of defining a model that doesn't fit with a particular >>> implementation or machine and get shot down when it's brought to >>> the forum. For example, you may remember the PERUSE performance >> interface >>> that assumed a model of MPI that implementers didn't approve, >>> because it didn't fit their implementation or was >>> difficult/expensive to support. Now, to replace PERUSE, we've got >>> the MPI_T interface which doesn't specify *anything* but appears >>> to be supported by the forum. >>> >>> I agree though that having more specific error information when >>> it's available would be very useful. You might consider taking an >>> approach similar to MPI_T -- allow MPI implementers to define any >>> specific error codes they can/want and then provide an interface >>> for decoding and interpreting the errors. >>> >>> Of course, this approach may not be useful for most applications >>> directly, but I imagine that a fault-tolerant MPI application or >>> a checkpoint/restart library could make use of the information, >>> assuming they could get at it. >>> >>> Kathryn >>> >>> >>> >>> >>> On 6/9/2011 8:20 AM, Howard Pritchard wrote: >>>> >>>> Hi Greg, >>>> >>>> I vote for an MPI-centric model. >>>> >>>> I also think that part of the job of MPI is to hide as much as >>>> possible things like 'exhaustion of network resources' and >>>> 'intermittent network failures'. Indeed, the very first >>>> sentence in section 2.8 says "MPI provides the user with >>>> reliable message transmission". >>>> >>>> The only reason the topic came up yesterday was in the context >>>> of the fail-stop model and what types of error codes might be >>>> returned by MPI before the official verdict was in that a >>>> fail-stop had occurred. Several of us checked what our >>>> implementations might do prior to that, and it could include >>>> returning MPI_ERR_OTHER. I could see how for someone writing a >>>> fault tolerant MPI application, something more useful than this >>>> rather ambiguous error code might be worth defining. >>>> >>>> Howard >>>> >>>> >>>> Bronevetsky, Greg wrote: >>>>> >>>>> I like the idea of having an abstract model of failures that >>>>> can approximate changes in system functionality due to >>>>> failures. However, I think before we go too far with this we >>>>> should consider the type of model we want to make. One option >>>>> is to make a system model that has as its basic elements >>>>> nodes, network links and other hardware components and >>>>> identifies points in time when stop functioning. The other >>>>> option is to make it MPI-centric by talking about the status >>>>> of ranks and point-to-point communication between them as >>>>> well as communicators and collective communication over them. >>>>> So in the first type of model we can talk about network >>>>> resource exhaustion and in the latter we can talk about an >>>>> intermittent >> inability to send messages over some or all communicators. >>>>> >>>>> I think that the MPI-centric model is a better option since >>>>> it talks exclusively about entities that exist in MPI and >>>>> ignores the physical phenomena that cause a given type of >>>>> degradation in >> functionality. >>>>> >>>>> The other question we need to discuss is the types of >>>>> problems we want to represent. We obviously care about >>>>> fail-stop failures but we're not talking about resource >>>>> exhaustion. Do we want to add error classes for transient >>>>> errors and if so, what about performance >> slowdowns? >>>>> >>>>> Greg Bronevetsky Lawrence Livermore National Lab (925) >>>>> 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com >>>>> >>>>> >>>>>> -----Original Message----- From: >>>>>> mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey Sent: >>>>>> Wednesday, June 08, 2011 11:36 AM To: MPI 3.0 Fault >>>>>> Tolerance and Dynamic Process Control working Group >>>>>> Subject: [Mpi3-ft] General Network Channel Failure >>>>>> >>>>>> It was mentioned in the conversation today that >>>>>> MPI_ERR_RANK_FAIL_STOP may not be the first error returned >>>>>> by an MPI call. In particular the MPI call may return an >>>>>> error symptomatic of a fail-stop process failure (e.g., >>>>>> network link failed - currently MPI_ERR_OTHER), before >>>>>> eventually diagnosing the event as a process failure. This >>>>>> 'space between' MPI_SUCCESS behavior and >>>>>> MPI_ERR_RANK_FAIL_STOP behavior is not currently defined, >>>>>> and probably should be for the application to cleanly move >>>>>> from set of semantics for one error class to another. >>>>>> >>>>>> The suggestion was to create a new general network error >>>>>> class (e.g., MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - >> MPI_ERR_COMM is >>>>>> taken) that can be returned when the operation cannot >>>>>> complete due to network issues (which might be later >>>>>> diagnosed as process failure and escalated to the >>>>>> MPI_ERR_RANK_FAIL_STOP semantics). >> You >>>>>> could also think about this error being used for network >>>>>> resource exhaustion as well (something that Tony mentioned >>>>>> during the last MPI Forum meeting). In which case retrying >>>>>> at a later time or taking some other action before trying >>>>>> again would be useful/expected. >>>>>> >>>>>> There are some issues with matching, and the implications >>>>>> on collective operations. If the network error is >>>>>> sticky/permanent then once the error is returned it will >>>>>> always be returned or escalated to fail-stop process >>>>>> failure (or more generally to a 'higher/more severe/more >>>>>> detailed' error class). A recovery proposal (similar to >>>>>> what we are developing for process failure) would allow the >>>>>> application to 'recover' the channel and continue >>>>>> communicating on it. >>>>>> >>>>>> >>>>>> The feeling was that this should be expanded into a full >>>>>> proposal, separate from the Run-Through Stabilization >>>>>> proposal. So we can continue with the RTS proposal, and >>>>>> bring this forward when it is ready. >>>>>> >>>>>> >>>>>> What to folks think about this idea? >>>>>> >>>>>> -- Josh >>>>>> >>>>>> -- Joshua Hursey Postdoctoral Research Associate Oak Ridge >>>>>> National Laboratory http://users.nccs.gov/~jjhursey >>>>>> _______________________________________________ mpi3-ft >>>>>> mailing list mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> _______________________________________________ mpi3-ft >>>>> mailing list mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>> _______________________________________________ mpi3-ft mailing >>> list mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> >> >> >> -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National >> Laboratory http://users.nccs.gov/~jjhursey >> >> _______________________________________________ mpi3-ft mailing >> list mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] General Network Channel Failure
by Bronevetsky, Greg 13 Jun '11

13 Jun '11

I like the idea of a query interface. This is primarily because I believe that fault tolerance should be the responsibility of middleware rather than applications. Writing scalable codes is already hard and writing your own reliability solutions that are also scalable and robust is almost certainly too much to ask for. However, if we leave the job to middleware, we'll need to provide an interface that is both portable and flexible. If we go for an MPI-level interface we get portability but we also get complexity because MPI provides developers with a large number of capabilities (send to/receive from a rank, collectively communicate on communicator, read/write a file, etc.). A full API will need to allow the application to perform queries to identify which capabilities are available after a given fault. A decent API for this might be to allow users to query for individual capabilities and also allow for short-cut APIs. For example, the run-through API focuses on fail-stop failures of one or more ranks. As such, if MPI reports that a rank has failed, this implies that a fixed set of capabilities is no longer available. We can extend this idea to other failure models that virtualize sets of capabilities under a single name, meaning that a problem with a single virtual capability (slow network or corrupted file) implies a specific change to large number of capabilities (all sends/receives are slow or reads/writes to just this file are invalid). Further, failure models themselves do not need to be specified in the spec. The reason this might make sense is if we want to model various levels using different abstractions. A simple way to represent network failures is to treat the network as a monolithic entity. In this model MPI_NETWORK_ERROR means that no communication can take place. A more precise abstraction might treat the network as a set of communication islands and one island can fail without causing errors for communication on other islands. An even more precise model may more individual rank-to-rank connections and allow the user to identify which connections are valid/invalid after a failure. All these models are just different portable approximations of the real system state and are useful for different applications. Right now I'm nervous about choosing one to be in the spec while leaving the others out. However, it should be pretty useful to force MPI to implement some fairly detailed choice and leave the higher-level ones to be implemented outside the spec but provide a standard interface to present them to users. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey > Sent: Monday, June 13, 2011 2:06 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] General Network Channel Failure > > Yeah the 'system model' is probably not going to fly with the MPI Forum. > Mostly because we would be in the business then of defining what a 'node', > 'NIC', 'RAM', etc. are - something the standard strives to not defined for > future proofing. > > So I like the MPI model a bit more. For resource exhaustion we do get a bit of > wiggle room with the 'reliable communication' requirement in > 3.7 "If the call causes some system resource to be exhausted, then it will fail > and return an error code." And for memory in 8.2 "The function > MPI_ALLOC_MEM may return an error code of class MPI_ERR_NO_MEM to > indicate it failed because memory is exhausted." > > So even though MPI shys away from defining network resources and > memory, we might be able to find some acceptable language within the > confines of these few error statements. So I think we can talk about > connections between pairs (P2P) and groups (collectives) of processes, and > maybe even connections to files (I/O). For each error class that we want to > define we need to go through the process of asking "so if this error code was > returned from this operation, what would that mean to the programmer? > How do we want them to react to it?" Taking resource exhaustion as an > example, for point-to-point messages if the application receives such an > error then they may want to repost the message later. But if the same error > is returned locally from a collective operation we need to determine if the > user should repost the same collective or if the collective is guaranteed to > return at all other processes with either success or some other error. > > I believe that the primitives that we specified for fail-stop process failure can > likely be extended to support other classes of errors without too much > modification. But we will have to see as we go along how true that becomes. > > > For Greg, this is the thread of conversation in which we will likely want to > explore transitent failures and possibly even revive the discussion about > performance degradation notification (probably performance notification is > best saved for another thread). But thinking about modern failure detectors > that provide an assurance range (accural failure detectors are what I am > thinking of > specifically) then the MPI may want to return an error indicating something > symptomatic of the eventual diagnosis of a process as either fail-stop or > transient. > > > To Kathryn's point about the MPI_T proposal, so you are suggesting that we > provide a query interface that the application can use to determine what > types of errors can be handled and how the MPI implementation allows the > application to continue after them? I think this is a good/necessary idea for > tools support - so maybe as an extension to the MPI_T proposal. But I am > hesitant to do so as the main mechanism for processing emerging errors. I > think that applications would struggle with something that flexible since, it > seems to imply that, they would have to adapt not to the error code, but to > the error code and reported capabilities of the MPI implementation. So for > the programmer they would need to have a check for the error code, a query > for capabilities, and then a switch statement for all the possible actions that > they can take. I like the flexibility, but I think it becomes too much of a > burden on the programmer. > > I guess the counter argument would be that the MPI implementations > should be, today, documenting all expected errors that they could return to > the user. The MPI_T-like interface just provides a programatic way to access > this information and react to it dynamically, instead of relying on > documentation updates and a round of software updates. So this would fit > nicely into the existing requirements for MPI implementations, though at > some additional complexity to the end user. > > Interesting idea, if I understand it correctly. What do others think? > > > -- Josh > > > On Sun, Jun 12, 2011 at 12:09 PM, Kathryn Mohror <kathryn(a)llnl.gov> wrote: > > Hi all, > > > > (I sent this earlier, but I don't think it went through because I sent > > it from the wrong email address. I apologize if you get an extra > > copy.) > > > > I know I haven't participated in this working group yet, so I may be > > missing some context, but I couldn't resist putting my two cents in! > > > > I think that an MPI-centric approach is best. Otherwise, you run the > > risk of defining a model that doesn't fit with a particular > > implementation or machine and get shot down when it's brought to the > > forum. For example, you may remember the PERUSE performance > interface > > that assumed a model of MPI that implementers didn't approve, because > > it didn't fit their implementation or was difficult/expensive to > > support. Now, to replace PERUSE, we've got the MPI_T interface which > > doesn't specify > > *anything* but appears to be supported by the forum. > > > > I agree though that having more specific error information when it's > > available would be very useful. You might consider taking an approach > > similar to MPI_T -- allow MPI implementers to define any specific > > error codes they can/want and then provide an interface for decoding > > and interpreting the errors. > > > > Of course, this approach may not be useful for most applications > > directly, but I imagine that a fault-tolerant MPI application or a > > checkpoint/restart library could make use of the information, assuming > > they could get at it. > > > > Kathryn > > > > > > > > > > On 6/9/2011 8:20 AM, Howard Pritchard wrote: > >> > >> Hi Greg, > >> > >> I vote for an MPI-centric model. > >> > >> I also think that part of the job of MPI is to hide as much as > >> possible things like 'exhaustion of network resources' > >> and 'intermittent network failures'. Indeed, the very first sentence > >> in section 2.8 says "MPI provides the user with reliable message > >> transmission". > >> > >> The only reason the topic came up yesterday was in the context of the > >> fail-stop model and what types of error codes might be returned by > >> MPI before the official verdict was in that a fail-stop had occurred. > >> Several of us checked what our implementations might do prior to > >> that, and it could include returning MPI_ERR_OTHER. I could see how > >> for someone writing a fault tolerant MPI application, something more > >> useful than this rather ambiguous error code might be worth defining. > >> > >> Howard > >> > >> > >> Bronevetsky, Greg wrote: > >>> > >>> I like the idea of having an abstract model of failures that can > >>> approximate changes in system functionality due to failures. > >>> However, I think before we go too far with this we should consider > >>> the type of model we want to make. One option is to make a system > >>> model that has as its basic elements nodes, network links and other > >>> hardware components and identifies points in time when stop > >>> functioning. The other option is to make it MPI-centric by talking > >>> about the status of ranks and point-to-point communication between > >>> them as well as communicators and collective communication over > >>> them. So in the first type of model we can talk about network > >>> resource exhaustion and in the latter we can talk about an intermittent > inability to send messages over some or all communicators. > >>> > >>> I think that the MPI-centric model is a better option since it talks > >>> exclusively about entities that exist in MPI and ignores the > >>> physical phenomena that cause a given type of degradation in > functionality. > >>> > >>> The other question we need to discuss is the types of problems we > >>> want to represent. We obviously care about fail-stop failures but > >>> we're not talking about resource exhaustion. Do we want to add error > >>> classes for transient errors and if so, what about performance > slowdowns? > >>> > >>> Greg Bronevetsky > >>> Lawrence Livermore National Lab > >>> (925) 424-5756 > >>> bronevetsky(a)llnl.gov > >>> http://greg.bronevetsky.com > >>> > >>> > >>>> -----Original Message----- > >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > >>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey > >>>> Sent: Wednesday, June 08, 2011 11:36 AM > >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >>>> Group > >>>> Subject: [Mpi3-ft] General Network Channel Failure > >>>> > >>>> It was mentioned in the conversation today that > >>>> MPI_ERR_RANK_FAIL_STOP may not be the first error returned by an > >>>> MPI call. In particular the MPI call may return an error > >>>> symptomatic of a fail-stop process failure (e.g., network link > >>>> failed - currently MPI_ERR_OTHER), before eventually diagnosing the > >>>> event as a process failure. This 'space between' MPI_SUCCESS > >>>> behavior and MPI_ERR_RANK_FAIL_STOP behavior is not currently > >>>> defined, and probably should be for the application to cleanly move > >>>> from set of semantics for one error class to another. > >>>> > >>>> The suggestion was to create a new general network error class > >>>> (e.g., MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - > MPI_ERR_COMM is > >>>> taken) that can be returned when the operation cannot complete due > >>>> to network issues (which might be later diagnosed as process > >>>> failure and escalated to the MPI_ERR_RANK_FAIL_STOP semantics). > You > >>>> could also think about this error being used for network resource > >>>> exhaustion as well (something that Tony mentioned during the last > >>>> MPI Forum meeting). In which case retrying at a later time or > >>>> taking some other action before trying again would be > >>>> useful/expected. > >>>> > >>>> There are some issues with matching, and the implications on > >>>> collective operations. If the network error is sticky/permanent > >>>> then once the error is returned it will always be returned or > >>>> escalated to fail-stop process failure (or more generally to a > >>>> 'higher/more severe/more detailed' error class). A recovery > >>>> proposal (similar to what we are developing for process > >>>> failure) > >>>> would allow the application to 'recover' the channel and continue > >>>> communicating on it. > >>>> > >>>> > >>>> The feeling was that this should be expanded into a full proposal, > >>>> separate from the Run-Through Stabilization proposal. So we can > >>>> continue with the RTS proposal, and bring this forward when it is > >>>> ready. > >>>> > >>>> > >>>> What to folks think about this idea? > >>>> > >>>> -- Josh > >>>> > >>>> -- > >>>> Joshua Hursey > >>>> Postdoctoral Research Associate > >>>> Oak Ridge National Laboratory > >>>> http://users.nccs.gov/~jjhursey > >>>> _______________________________________________ > >>>> mpi3-ft mailing list > >>>> mpi3-ft(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >>> > >>> _______________________________________________ > >>> mpi3-ft mailing list > >>> mpi3-ft(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >> > >> > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] The state of MPI is undefined
by Darius Buntinas 13 Jun '11

13 Jun '11

On Jun 13, 2011, at 3:36 PM, Josh Hursey wrote: > On Mon, Jun 13, 2011 at 12:36 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> >> I'm not sure I like the wording: >> >> If the MPI implementation can continue operating after process failure >> then it must return an appropriate error class (e.g., >> MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined >> in Chapter 17. >> >> I don't think we can say that "if MPI returns error code X" ==> "MPI state is defined," because if some state corrupting error occurs, the state of MPI is undefined and the implementation might return error code X. Rather, I think we should use the form "if error FOO occurs" ==> "MPI state will be BAR." So for example, we can say "if an unrecoverable communication error occurs, the MPI implementation will return a MPI_ERR_COMMUNICATION for all pending communication operations and blah, blah, blah." > > For the case where the MPI was corrupted so that it is forced to > return a particular error code, I do not know how we can get around > the fact that we cannot define the state of MPI (the error forced the > MPI implementation to lie to the application). So even in the form "if > error FOO occurs" ==> "MPI state will be BAR" the 'will be' is too > strong since the corruption forced the return of FOO without the > consent of the MPI implementation. But if we say 'might be' then we > are back to introducing a proposal that cannot be used by applications > - lack of solid semantics. Sorry, I should have been more specific. When I say "if error FOO occurs" I don't mean error FOO is returned but rather that an error condition occurred. E.g., if a communication error occurs, an MPI_ERR_COMMUNICATION error will be returned." So this explicitly excludes the state corrupting error. Note that it's still possible that a state corrupting error results in MPI_ERR_COMMUNICATION to be returned, but we can't prevent that case. But at least we can say that. e.g., assuming the state of the library is not already corrupted, for a compliant implementation, a communication error (however we define that) will not corrupt the library and the library will behave as prescribed. > > So maybe we should add an advice to users that states that: > ------------ > It is possible that the state of MPI becomes corrupted in an > undetectable manner. This undetectable error might force the MPI > implementation to return an error code (or even success) > unintentionally. For this pathological case there exist no mechanisms > to determine the correct running state. A high-quality implementation > will strive to return the correct return code from all MPI operations. > ------------ I think this is OK, but maybe we can make it more general: It is possible that some error/fault* may cause the state of MPI to become corrupted in an undetectable manner. The behavior of MPI in this case is undefined, and it is possible that the implementation returns incorrect error codes (or even MPI_SUCCESS). So while a high-quality implementation will strive to always return correct return codes from MPI operations, it may not be possible in all cases. * Perhaps we should define error and fault and check for correct usage in the standard. Does anyone have good definitions? Is an error a detected (at some level) fault? Should we drop error and just use fault? -d > >> >> A blanket statement we can make about errors, then, would be something like "Unless otherwise noted in the standard, after an error is detected, the state of MPI is undefined." >> >> Another thought, we may get resistance from the Forum about the cost of checking for faults. Maybe we discussed this already, I can't remember. Do we want to make FT optional, and say "If the MPI implementation supports FT, then the behavior will be blah"? > > How about: > ------------------ > MPI does not provide the user with transparent process recovery upon > process failure. > Once a process fails, MPI does not guarantee that the job can continue > or, if the job can continue, that the process can be recovered. > If a process failure occurs, and the MPI implementation is able to > correctly continue operating after process failure then the MPI > implementation will return an appropriate error class (e.g., > MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined > in Chapter 17. > The MPI implementation documentation will provide information on the > possible effect of each supported class of errors. > > Advice to users: > It is possible that the state of MPI becomes corrupted in an > undetectable manner. This undetectable error might force the MPI > implementation to return an error code (or even success) > unintentionally. For this pathological case there exist no mechanisms > to determine the correct running state. A high-quality implementation > will strive to return the correct return code from all MPI operations. > ------------------ > > I like the idea of keeping the softer language with regard to whether > or not it is required, but instead say that if the user sees error > code X (in this case MPI_ERR_RANK_FAIL_STOP) then they can expect the > behavior outlined in Chapter 17. If they do not see that error, but > something else then it is implementation defined what will happen > afterwards. And the corner case is described to the users, which is an > understandable limitation of any software system. > > > What do folks think about this? > > -- Josh > > >> >> -d >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] General Network Channel Failure
by Josh Hursey 13 Jun '11

13 Jun '11

Yeah the 'system model' is probably not going to fly with the MPI Forum. Mostly because we would be in the business then of defining what a 'node', 'NIC', 'RAM', etc. are - something the standard strives to not defined for future proofing. So I like the MPI model a bit more. For resource exhaustion we do get a bit of wiggle room with the 'reliable communication' requirement in 3.7 "If the call causes some system resource to be exhausted, then it will fail and return an error code." And for memory in 8.2 "The function MPI_ALLOC_MEM may return an error code of class MPI_ERR_NO_MEM to indicate it failed because memory is exhausted." So even though MPI shys away from defining network resources and memory, we might be able to find some acceptable language within the confines of these few error statements. So I think we can talk about connections between pairs (P2P) and groups (collectives) of processes, and maybe even connections to files (I/O). For each error class that we want to define we need to go through the process of asking "so if this error code was returned from this operation, what would that mean to the programmer? How do we want them to react to it?" Taking resource exhaustion as an example, for point-to-point messages if the application receives such an error then they may want to repost the message later. But if the same error is returned locally from a collective operation we need to determine if the user should repost the same collective or if the collective is guaranteed to return at all other processes with either success or some other error. I believe that the primitives that we specified for fail-stop process failure can likely be extended to support other classes of errors without too much modification. But we will have to see as we go along how true that becomes. For Greg, this is the thread of conversation in which we will likely want to explore transitent failures and possibly even revive the discussion about performance degradation notification (probably performance notification is best saved for another thread). But thinking about modern failure detectors that provide an assurance range (accural failure detectors are what I am thinking of specifically) then the MPI may want to return an error indicating something symptomatic of the eventual diagnosis of a process as either fail-stop or transient. To Kathryn's point about the MPI_T proposal, so you are suggesting that we provide a query interface that the application can use to determine what types of errors can be handled and how the MPI implementation allows the application to continue after them? I think this is a good/necessary idea for tools support - so maybe as an extension to the MPI_T proposal. But I am hesitant to do so as the main mechanism for processing emerging errors. I think that applications would struggle with something that flexible since, it seems to imply that, they would have to adapt not to the error code, but to the error code and reported capabilities of the MPI implementation. So for the programmer they would need to have a check for the error code, a query for capabilities, and then a switch statement for all the possible actions that they can take. I like the flexibility, but I think it becomes too much of a burden on the programmer. I guess the counter argument would be that the MPI implementations should be, today, documenting all expected errors that they could return to the user. The MPI_T-like interface just provides a programatic way to access this information and react to it dynamically, instead of relying on documentation updates and a round of software updates. So this would fit nicely into the existing requirements for MPI implementations, though at some additional complexity to the end user. Interesting idea, if I understand it correctly. What do others think? -- Josh On Sun, Jun 12, 2011 at 12:09 PM, Kathryn Mohror <kathryn(a)llnl.gov> wrote: > Hi all, > > (I sent this earlier, but I don't think it went through because I sent it > from the wrong email address. I apologize if you get an extra copy.) > > I know I haven't participated in this working group yet, so I may be > missing some context, but I couldn't resist putting my two cents in! > > I think that an MPI-centric approach is best. Otherwise, you run the > risk of defining a model that doesn't fit with a particular > implementation or machine and get shot down when it's brought to the > forum. For example, you may remember the PERUSE performance interface > that assumed a model of MPI that implementers didn't approve, because it > didn't fit their implementation or was difficult/expensive to support. Now, > to replace PERUSE, we've got the MPI_T interface which doesn't specify > *anything* but appears to be supported by the forum. > > I agree though that having more specific error information when it's > available would be very useful. You might consider taking an approach > similar to MPI_T -- allow MPI implementers to define any specific error > codes they can/want and then provide an interface for decoding and > interpreting the errors. > > Of course, this approach may not be useful for most applications > directly, but I imagine that a fault-tolerant MPI application or ï¿½a > checkpoint/restart library could make use of the information, assuming > they could get at it. > > Kathryn > > > > > On 6/9/2011 8:20 AM, Howard Pritchard wrote: >> >> Hi Greg, >> >> I vote for an MPI-centric model. >> >> I also think that part of the job of MPI is to hide as much >> as possible things like 'exhaustion of network resources' >> and 'intermittent network failures'. ï¿½Indeed, the very first >> sentence in section 2.8 says "MPI provides the user with >> reliable message transmission". >> >> The only reason the topic came up yesterday was in the >> context of the fail-stop model and what types of error >> codes might be returned by MPI before the official >> verdict was in that a fail-stop had occurred. ï¿½Several of >> us checked what our implementations might do prior to >> that, and it could include returning MPI_ERR_OTHER. ï¿½I >> could see how for someone writing a fault tolerant MPI >> application, something more useful than this rather ambiguous >> error code might be worth defining. >> >> Howard >> >> >> Bronevetsky, Greg wrote: >>> >>> I like the idea of having an abstract model of failures that can >>> approximate changes in system functionality due to failures. However, I >>> think before we go too far with this we should consider the type of model we >>> want to make. One option is to make a system model that has as its basic >>> elements nodes, network links and other hardware components and identifies >>> points in time when stop functioning. The other option is to make it >>> MPI-centric by talking about the status of ranks and point-to-point >>> communication between them as well as communicators and collective >>> communication over them. So in the first type of model we can talk about >>> network resource exhaustion and in the latter we can talk about an >>> intermittent inability to send messages over some or all communicators. >>> >>> I think that the MPI-centric model is a better option since it talks >>> exclusively ï¿½about entities that exist in MPI and ignores the physical >>> phenomena that cause a given type of degradation in functionality. >>> >>> The other question we need to discuss is the types of problems we want to >>> represent. We obviously care about fail-stop failures but we're not talking >>> about resource exhaustion. Do we want to add error classes for transient >>> errors and if so, what about performance slowdowns? >>> >>> Greg Bronevetsky >>> Lawrence Livermore National Lab >>> (925) 424-5756 >>> bronevetsky(a)llnl.gov >>> http://greg.bronevetsky.com >>> >>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>>> Sent: Wednesday, June 08, 2011 11:36 AM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: [Mpi3-ft] General Network Channel Failure >>>> >>>> It was mentioned in the conversation today that MPI_ERR_RANK_FAIL_STOP >>>> may not be the first error returned by an MPI call. In particular the >>>> MPI call >>>> may return an error symptomatic of a fail-stop process failure (e.g., >>>> network >>>> link failed - currently MPI_ERR_OTHER), before eventually diagnosing the >>>> event as a process failure. This 'space between' MPI_SUCCESS behavior >>>> and >>>> MPI_ERR_RANK_FAIL_STOP behavior is not currently defined, and probably >>>> should be for the application to cleanly move from set of semantics for >>>> one >>>> error class to another. >>>> >>>> The suggestion was to create a new general network error class (e.g., >>>> MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - MPI_ERR_COMM is >>>> taken) that can be returned when the operation cannot complete due to >>>> network issues (which might be later diagnosed as process failure and >>>> escalated to the MPI_ERR_RANK_FAIL_STOP semantics). You could also think >>>> about this error being used for network resource exhaustion as well >>>> (something that Tony mentioned during the last MPI Forum meeting). In >>>> which case retrying at a later time or taking some other action before >>>> trying >>>> again would be useful/expected. >>>> >>>> There are some issues with matching, and the implications on collective >>>> operations. If the network error is sticky/permanent then once the error >>>> is >>>> returned it will always be returned or escalated to fail-stop process >>>> failure (or >>>> more generally to a 'higher/more severe/more detailed' error class). A >>>> recovery proposal (similar to what we are developing for process >>>> failure) >>>> would allow the application to 'recover' the channel and continue >>>> communicating on it. >>>> >>>> >>>> The feeling was that this should be expanded into a full proposal, >>>> separate >>>> from the Run-Through Stabilization proposal. So we can continue with the >>>> RTS proposal, and bring this forward when it is ready. >>>> >>>> >>>> What to folks think about this idea? >>>> >>>> -- Josh >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] The state of MPI is undefined
by Howard Pritchard 13 Jun '11

13 Jun '11

Hi Josh, > > How about: > ------------------ > MPI does not provide the user with transparent process recovery upon > process failure. > Once a process fails, MPI does not guarantee that the job can continue > or, if the job can continue, that the process can be recovered. > If a process failure occurs, and the MPI implementation is able to > correctly continue operating after process failure then the MPI > implementation will return an appropriate error class (e.g., > MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined > in Chapter 17. > The MPI implementation documentation will provide information on the > possible effect of each supported class of errors. > > Advice to users: > It is possible that the state of MPI becomes corrupted in an > undetectable manner. This undetectable error might force the MPI > implementation to return an error code (or even success) > unintentionally. For this pathological case there exist no mechanisms > to determine the correct running state. A high-quality implementation > will strive to return the correct return code from all MPI operations. > ------------------ I vote for this wording. Howard -- Howard Pritchard Software Engineering Cray, Inc.

1 0