mpiwg-ft

Download

[email protected]

October 2011

7 participants
45 discussions

Re: [Mpi3-ft] Latest version of chapter
by Supalov, Alexander 24 Oct '11

24 Oct '11

Thanks. See below. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Saturday, October 22, 2011 6:26 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Latest version of chapter Alexander, Thanks for the comments. It is helping me, at least, think through some of the use cases in a bit more detail. I think I'm ok with keeping the inheritance of the FailHandler matching the inheritance of the ErrHandler. So in some, but not all, cases it is inherited, and we don't create a special inheritance rule for the FailHandler different from the ErrHandler. AS> OK. What are the exceptions? There is a bit of a race between firing of a FailHanlder and the destruction of the communication object. I think to some degree that is unavoidable, and the user will need to have setup an FailHandler on another communicator to get the notification. If the failed process is known at the top of the destruction function, then it might be called at the top before destruction. Otherwise it is as if the process failed just after the destruction. So we never get in the case where the FailHandler is triggered with an invalid communicator. Humm... the problem then is that if some processes destroyed the communicator while others are in the FailHandler with the not-yet-destroyed communicator, the user will have to somehow know that the communicator object, though locally valid, is not globally valid and might not be able to be used for communication. Icky... My gut reaction is that MPI should not put the user into this situation, but that means the the comm_free operation is not just collective, but synchronizing. I don't think we want to allow the user to destroy the communication object in the FailHandler (or even ErrHandler). That might get tricky to support. AS> I thought about this, too. There actually two races here: 1. It appears that unless there are two or more conflicting handlers in action trying to recreate the failed processes, the set of the failed processes of a communicator may only increase between the call and the moment the handler deals with the failures. So, we should probably not be afraid of this race condition, but we should make provisions for the handler to be fired up once again at a later occasion for the additional failed processes if there were any. We should also put the onus on the user of not writing mutually conflicting handlers. 2. The race that you mentioned. I have to think about it more thoroughly. The delegation of the object destruction to the non-handler part of user code seems a good first step, but it somewhat limits the value of the handler, relegating it to a mere event registering facility rather than a problem resolution agent. I'm good with the notion that the FailHandler is fired once for every communicator that includes the failed process. So if Process X is in both commA and commB and those communicators have FailHandler fhA and fhB registered, respectively. Then the failure of process X would trigger a call to fhA (the next time commA is used) -and- fhB (the next time commB is used). That way we make sure that every communicator that wanted to see the error is able to see it (for library support). And the application is responsible for coordinating recovery between multiple FailHandler callbacks that they might see due to the use of multiple communicators at the application level. AS> OK. The last point seems to be do we fire the per-communicator FailHanlder once for each process failure or once for a set of process failures. I think scalability would push us towards the set of process failures, particularly if we are starting to talk about nodes with 100s of processes so losing a node might trigger a lot of callbacks x the number of communicators including those processes. So I am leaning towards the firing of the FailHandler once for a set of failures. Maybe we can adjust the signature of the FailHandler function to return a locally consistent MPI_Group of failed processes (similar to MPI_Comm_group_failed) that represent the view of failed processes on the communicator at the time of the FailHandler call. AS> Yes, having some consolidated information on the failed processes would indeed help in case of failure handler call aggregation. Actually, an array of MPI_Status objects might do as well. There is the issue that, what if a process fails while the application is in the FailHandler. In particular if we allow the user to use MPI functions in the FailHandler, the new failure might trigger the FailHandler to be called again. The user would have to make sure that the two (or N) FailHandler invocations do not interfere with one another - which might get tricky with message matching if they are using MPI functions. I guess we could have the restriction that the FailHandler would -not- be triggered from MPI functions used within the FailHandler (or only reenable it after the FailHandler returns). AS> Re-entrance is always tricky. May we also ask the user to write the handlers in a mutually exclusive manner? So that they at least a not stacked recursively by chance. Might somewhat complicate progress tracking, though. I'll take a pass at the document today and include these semantic adjustments. I suspect the semantics will continue to be refined based on further discussion, but it will give us a bit of a straw man to work against as we keep taking through this use case in the coming weeks. Thanks, Josh On Sat, Oct 22, 2011 at 12:18 AM, Supalov, Alexander <alexander.supalov(a)intel.com> wrote: > PS. Small extension of the summary at the bottom of my message below, to cross all t's: "... should fire the resp. _inheritable_ failure handlers ..." > > -----Original Message----- > From: Supalov, Alexander > Sent: Saturday, October 22, 2011 6:01 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: RE: [Mpi3-ft] Latest version of chapter > > Hi, > > I think what emerges is an approach to a good solution. There are some loose ends, though. Going thru the discussion: > >>>>> [Darius] The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. > > Let's clarify this. Imagine I had commA with some process X in it. X failed when commA was being destroyed. I did not get the notification in the resp. call on my process Y because it had already returned there. Later I, being totally unaware of the process X failure, try to create a new commB with process X in it. Will I notice that it's gone? How will I detect that now technically as an MPI implementor? Probably, I'll track process state deep inside the library. How will I detect that as an MPI user - when my handlerA is gone? Probably, I want handlerB, if available, to be called at this moment. I hope this example is reasonable as it partially underpins the remaining comments below. > >>>> [Josh] A few things that probably should be clarified with the new FailHandler: >>>> - Is it inherited by new communicators like other error handlers? >>> >>> [Darius] I'd say no. Because all it would do is call the same handler once for every communicator for the failure of the same process. >> >>[Josh] I agree. > > Why this exception? A handler can trivially be made smart enough to notice it's being called over and over again in reaction to the death of one and the same process X. Actually, it does not need to be smart at all: when called, it will deal with the resp. communicator/file/win. It's the library that needs to be smart to mark process X failure only once and make sure all resp. communicators get notified in due time. > > Moreover, if inheritance is disabled in this case, we're more likely to get wrong programs. People will just keep forgetting about this special feature of the failure handlers, and fail in an uncontrolled fashion despite their best effort - a few weeks into the run. > > By excluding the inheritance here we create an exception to the rule that should be justified by more than our assumption of what the handler will do I think. > >>>> [Josh] If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. > > Again, this is not something I'd do. We may force the user to react to many process failures in one handler call. This is not what they may want to do. It's easier to write a handler that deals with one process failure at a time, and let the library care of invoking it in the relatively rare case of multiple process failures. Even if we think about multi- and manycore cases, the number of failed processes will normally be limited by the number of cores per node, which is not going to be astronomic for quite a while I guess. I.e., we don't seem to be creating a scalability bottleneck here. > > If we do care about scalability even here, we should rather give the user a way to tell the library: "Yeah, I've dealt with this dead process X you've just reported. By the way, I've also dealt with processes Y1 to Yn you haven't reported yet, but you'll understand, right?" Then, and only then may the library skip notifying the user of the process Y1 thru Yn failures on the same comm/file/win . It still has to report all failures on other comm/file/win, though. > > To simplify this, the library might report not one but a list of failures in the failure handler, or at least their total number for the moment of the handler invocation. Then the user will be well equipped to handle them all in one go. If he misses one, by chance or by sloppiness or by race condition, he'll get a shout (or shouts) next time. > > I'm not as deep into this proposal to judge at the moment if this case is already covered, so, take this as a user request. They will mostly be rather naive about MPI FT at first, but they will have healthy instincts, common sense, and some MPI experience for sure. The above treatise is a reflection on what I think they will expect of us on this backdrop. > >> [Josh] Thinking a bit about the implementation it should not be too bad to >> track such things. We could keep a boolean (or do some fun function >> pointer hacking) on the communicator that is flipped whenever a new >> failure is detected, then flip it back after firing the error handler. >> Similar boolean to what we might use to disable collectives. > > Looking above and summarizing, we probably need to track a process status first and once somewhere inside the implementation. When one process fails, all resp. communicators should be marked as potentially problematic, or another, more scalable mechanism should be used in reactive, on-demand fashion. Then the resp. calls on them should fire the resp. failure handlers at appropriate times (pt2pt op w/ the failed process, a collective op invocation, etc.) w/o any assumptions as to how much the user will want to do there. The users should in turn have a way to tell the library that they've dealt with more than just the immediately reported failure(s) that caused the handler to be invoked in the first place. > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey > Sent: Saturday, October 22, 2011 12:20 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] Latest version of chapter > > On Fri, Oct 21, 2011 at 6:01 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> >> On Oct 21, 2011, at 4:18 PM, Josh Hursey wrote: >> >>> The original problem was that the application wanted uniform >>> notification of a process failure (restricted to the set of processes >>> in the group associated with the communication objection). The current >>> error handlers are only fired when interacting with the failed process >>> directly (P2P) or indirectly (ANY_SOURCE, collectives). >>> >>> The requesters (who I believe are on the list and may want to pipe up) >>> were ok with having the callback triggered at an MPI boundary - so not >>> really asynchronous just not associated with the call. >>> >>> Maybe it is enough to restrict the notification to operations on the >>> communicator. So the FailHandler registered on commA is only fired >>> when commA is being used. The application would have to register the >>> FailHandler on all communicators that it is using and wants >>> notification from. But that would preserve some separation between the >>> library and application. >> >> I think that makes sense. > > Why don't we start with that restriction, and run it by folks next week. > >> >>> >>> A few things that probably should be clarified with the new FailHandler: >>> - Is it inherited by new communicators like other error handlers? >> >> I'd say no. Because all it would do is call the same handler once for every communicator for the failure of the same process. > > I agree. > >> >>> - Without the communicator scope restriction mentioned above, if a >>> process fails, does it fire all of the FailHanders registered on >>> communication objects containing that process? (I think yes) If so, we >>> should probably state that we do not guarantee any ordering of these >>> calls. >> >> If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. > > > That sounds good to me. > > Thinking a bit about the implementation it should not be too bad to > track such things. We could keep a boolean (or do some fun function > pointer hacking) on the communicator that is flipped whenever a new > failure is detected, then flip it back after firing the error handler. > Similar boolean to what we might use to disable collectives. > >> >>> - In the function signatures the errhandler is 'int'/'integer' and >>> should probably be MPI_Errhandler or similar handle. >> >> I copied the prototypes from the error handler section. > > In section 8.3.1 of MPI 2.2 they are of the type MPI_Errhandler. So > pointers to the error handler function prototype. We can probably use > the same function pointer signature and object for these new > functions. > >> >>> - On the topic of what functions you can use inside, we can probably >>> use the language from the error handlers. I think it allows the user >>> to do pretty much anything they want, though I'd have to double check. >>> It might be that the standard is silent on this point, so no specific >>> restrictions are defined. >> >> I didn't see any restrictions, but then the standard says that all bets are off when you get an error, so calling anything at that point is undefined. > > I think staying silent for now is a good idea. But maybe we can think > about it over the weekend and talk more about it next week. > > > Darius: Do you have some time to make some of these changes to the > chapter and post a new copy of the document to the ticket? We probably > want to whole MPI standard text since some text changed outside of the > chapter for MPI_Finalize stuff. If not, I can probably get to it late > this evening, or tomorrow. > > > Thanks, > Josh > > >> >> -d >> >> >>> >>> >>> -- Josh >>> >>> >>> On Fri, Oct 21, 2011 at 4:54 PM, Supalov, Alexander >>> <alexander.supalov(a)intel.com> wrote: >>>> Imagine I use some data protection scheme inside B. I won't be affected by "wrong" libraries that I call before or after my protection is on. I may be affected by an asynchronous call out of "another world" that is possible if handlerA is called from within my library B. I.e., in the sequence >>>> >>>> A-B-A >>>> >>>> this extension allows B to be "hacked" by A by just killing one process at the right time. Moreover, I can clean up the callbacks by using MPI_Comm_create instead of MPI_Comm_dup. I cannot prevent an asynchronous handler from being called. >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>> Sent: Friday, October 21, 2011 10:44 PM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>> >>>> >>>> On Oct 21, 2011, at 3:19 PM, Supalov, Alexander wrote: >>>> >>>>> Thanks. See below (prefix "AS>"). >>>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>> Sent: Friday, October 21, 2011 9:57 PM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>> >>>>> >>>>> Let's say you have commA used by library A and commB used by library B. >>>>> Library A has registered the proc failure handler called handlerA on commA. >>>>> >>>>> Now, let's say a process that's in commA but not commB failed, and the thread is executing in library B and calls, e.g., MPI_Send(..., commB). >>>>> >>>>> The MPI implementation performs the MPI_Send operation normally, then calls handlerA(commA, MPI_ERR_PROC_FAIL_STOP), and returns from MPI_Send normally. >>>>> >>>>> While in handlerA, the subject communicator (commA) is passed as a parameter, so it won't be out of scope. >>>>> >>>>> Is it a problem that library A's handler is called from "within" library B? >>>>> >>>>> AS> Sure. This handler may have been written by someone else who does not know me or my B or anything else. I may not even want it to be called from within my library B for security reasons. What if it unwinds the stack, connects to A's HQ, and dumps my confidential memory all over there? >>>> >>>> Yikes! Don't link with libraries you don't trust :-) >>>> >>>> I don't know how to handle this case, but does the current standard prevent a library from snooping memory from other libraries? A library could set an attribute with a copy callback function on comm_world. That would be called from within another library's stack if that library tries to dup comm_world. >>>> >>>>> Moreover, by the time it's called, both A and commA may be the thing of the times long gone together with the context in which handlerA was supposed to be executed. What will it try to handle then and under what assumptions? I don't know. You? >>>> >>>> The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. >>>> >>>> -d >>>> >>>> >>>>> -d >>>>> >>>>> >>>>> On Oct 21, 2011, at 2:42 PM, Supalov, Alexander wrote: >>>>> >>>>>> Not really. How do you want the user make sense of that? E.g., I call A on commA, fail on commA asynchronously while calling a totally unrelated B on commB that has no failures in it, and am kicked out of B into someone else's error handler saying some "A" on "comma" failed? And what now? I may even have A and commA out of scope by then, possibly forever. >>>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>>> Sent: Friday, October 21, 2011 9:35 PM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>> >>>>>> With (regular) error handlers, they'll be called from within the function that raises the error. With failure notification, because they're being called as a result of an external event (process failure), you could be called from within any function, even one not related to the comm/file/win that you registered the process failure notification handler on. >>>>>> >>>>>> Does that make sense? >>>>>> >>>>>> -d >>>>>> >>>>>> On Oct 21, 2011, at 1:51 PM, Sur, Sayantan wrote: >>>>>> >>>>>>> 17.5.1:11-12 - "The error handler function will be called by the MPI implementation from within the context of some MPI function that was called by the user." >>>>>>> >>>>>>> Maybe we should that error handlers are called from MPI functions that are associated with that comm/file/win? >>>>>>> >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>>>>>>> Sent: Friday, October 21, 2011 10:28 AM >>>>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>>>> >>>>>>>> I just wanted to note that we want to distribute a copy of this >>>>>>>> chapter to the MPI Forum before the meeting. As such we are planning >>>>>>>> on sending out a copy at COB today (so Friday ~5:00 pm EDT) so that >>>>>>>> people have an opportunity to look at the document before the Monday >>>>>>>> plenary. So please send any edits or comments before COB today, so we >>>>>>>> can work them into the draft. >>>>>>>> >>>>>>>> We will post the draft to the ticket, so that people know where to >>>>>>>> look for the current draft. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Josh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 20, 2011 at 7:06 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Here's the latest version of the FT chapter is on the wiki (it's in a >>>>>>>> new location on the main FT page under "ticket #276". Please have a >>>>>>>> look and comment. >>>>>>>>> >>>>>>>>> Here's a direct link to the PDF: >>>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/raw- >>>>>>>> attachment/wiki/FaultToleranceWikiPage/ft.pdf >>>>>>>>> >>>>>>>>> Here's a summary of the changes Josh and I made: >>>>>>>>> >>>>>>>>> * Minor wording touchups >>>>>>>>> * Added new semantic for MPI_ANY_SOURCE with the >>>>>>>> MPI_ERR_ANY_SOURCE_DISABLED error code >>>>>>>>> * Coverted wording for all comm, win, fh creation operations to not >>>>>>>> require collectively active communicators (eliminate requirement for >>>>>>>> synchronization) >>>>>>>>> * Added missing reader_lock to ANY_SOURCE example >>>>>>>>> * Added case for MPI_WIN_TEST >>>>>>>>> >>>>>>>>> and >>>>>>>>> >>>>>>>>> One-sided section >>>>>>>>> clarified that window creation need not be blocking >>>>>>>>> clarified that RMA ops might not complete correctly even if >>>>>>>>> synchronization ops complete without error due to process >>>>>>>>> failures >>>>>>>>> Process failure notification >>>>>>>>> Added section describing new functions to add callbacks to comms, >>>>>>>>> wins and files that are called when proc failure is detected >>>>>>>>> Other wordsmithing/cleanup changes >>>>>>>>> >>>>>>>>> -d >>>>>>>>> _______________________________________________ >>>>>>>>> mpi3-ft mailing list >>>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Joshua Hursey >>>>>>>> Postdoctoral Research Associate >>>>>>>> Oak Ridge National Laboratory >>>>>>>> http://users.nccs.gov/~jjhursey >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> -------------------------------------------------------------------------------------- >>>>>> Intel GmbH >>>>>> Dornacher Strasse 1 >>>>>> 85622 Feldkirchen/Muenchen, Deutschland >>>>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>>>> Registergericht: Muenchen HRB 47456 >>>>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> -------------------------------------------------------------------------------------- >>>>> Intel GmbH >>>>> Dornacher Strasse 1 >>>>> 85622 Feldkirchen/Muenchen, Deutschland >>>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>>> Registergericht: Muenchen HRB 47456 >>>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> -------------------------------------------------------------------------------------- >>>> Intel GmbH >>>> Dornacher Strasse 1 >>>> 85622 Feldkirchen/Muenchen, Deutschland >>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>> Registergericht: Muenchen HRB 47456 >>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>> >>> >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -------------------------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen, Deutschland > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 > Ust.-IdNr./VAT Registration No.: DE129385895 > Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

[Mpi3-ft] New run-through stabilization draft available
by Josh Hursey 22 Oct '11

22 Oct '11

I just posted a new draft of the run-through stabilization proposal to the ticket and wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/FaultTolerance… In addition to the changes Darius previously mentioned, this draft includes some edits to the process failure handler, per the current state of discussion on the list. I suspect that further revisions will be necessary, but this draft will give the MPI Forum something to point to on Monday when we discuss some of the new revisions during the plenary session. Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Latest version of chapter
by Josh Hursey 22 Oct '11

22 Oct '11

Alexander, Thanks for the comments. It is helping me, at least, think through some of the use cases in a bit more detail. I think I'm ok with keeping the inheritance of the FailHandler matching the inheritance of the ErrHandler. So in some, but not all, cases it is inherited, and we don't create a special inheritance rule for the FailHandler different from the ErrHandler. There is a bit of a race between firing of a FailHanlder and the destruction of the communication object. I think to some degree that is unavoidable, and the user will need to have setup an FailHandler on another communicator to get the notification. If the failed process is known at the top of the destruction function, then it might be called at the top before destruction. Otherwise it is as if the process failed just after the destruction. So we never get in the case where the FailHandler is triggered with an invalid communicator. Humm... the problem then is that if some processes destroyed the communicator while others are in the FailHandler with the not-yet-destroyed communicator, the user will have to somehow know that the communicator object, though locally valid, is not globally valid and might not be able to be used for communication. Icky... My gut reaction is that MPI should not put the user into this situation, but that means the the comm_free operation is not just collective, but synchronizing. I don't think we want to allow the user to destroy the communication object in the FailHandler (or even ErrHandler). That might get tricky to support. I'm good with the notion that the FailHandler is fired once for every communicator that includes the failed process. So if Process X is in both commA and commB and those communicators have FailHandler fhA and fhB registered, respectively. Then the failure of process X would trigger a call to fhA (the next time commA is used) -and- fhB (the next time commB is used). That way we make sure that every communicator that wanted to see the error is able to see it (for library support). And the application is responsible for coordinating recovery between multiple FailHandler callbacks that they might see due to the use of multiple communicators at the application level. The last point seems to be do we fire the per-communicator FailHanlder once for each process failure or once for a set of process failures. I think scalability would push us towards the set of process failures, particularly if we are starting to talk about nodes with 100s of processes so losing a node might trigger a lot of callbacks x the number of communicators including those processes. So I am leaning towards the firing of the FailHandler once for a set of failures. Maybe we can adjust the signature of the FailHandler function to return a locally consistent MPI_Group of failed processes (similar to MPI_Comm_group_failed) that represent the view of failed processes on the communicator at the time of the FailHandler call. There is the issue that, what if a process fails while the application is in the FailHandler. In particular if we allow the user to use MPI functions in the FailHandler, the new failure might trigger the FailHandler to be called again. The user would have to make sure that the two (or N) FailHandler invocations do not interfere with one another - which might get tricky with message matching if they are using MPI functions. I guess we could have the restriction that the FailHandler would -not- be triggered from MPI functions used within the FailHandler (or only reenable it after the FailHandler returns). I'll take a pass at the document today and include these semantic adjustments. I suspect the semantics will continue to be refined based on further discussion, but it will give us a bit of a straw man to work against as we keep taking through this use case in the coming weeks. Thanks, Josh On Sat, Oct 22, 2011 at 12:18 AM, Supalov, Alexander <alexander.supalov(a)intel.com> wrote: > PS. Small extension of the summary at the bottom of my message below, to cross all t's: "... should fire the resp. _inheritable_ failure handlers ..." > > -----Original Message----- > From: Supalov, Alexander > Sent: Saturday, October 22, 2011 6:01 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: RE: [Mpi3-ft] Latest version of chapter > > Hi, > > I think what emerges is an approach to a good solution. There are some loose ends, though. Going thru the discussion: > >>>>> [Darius] The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. > > Let's clarify this. Imagine I had commA with some process X in it. X failed when commA was being destroyed. I did not get the notification in the resp. call on my process Y because it had already returned there. Later I, being totally unaware of the process X failure, try to create a new commB with process X in it. Will I notice that it's gone? How will I detect that now technically as an MPI implementor? Probably, I'll track process state deep inside the library. How will I detect that as an MPI user - when my handlerA is gone? Probably, I want handlerB, if available, to be called at this moment. I hope this example is reasonable as it partially underpins the remaining comments below. > >>>> [Josh] A few things that probably should be clarified with the new FailHandler: >>>> - Is it inherited by new communicators like other error handlers? >>> >>> [Darius] I'd say no. ï¿½Because all it would do is call the same handler once for every communicator for the failure of the same process. >> >>[Josh] I agree. > > Why this exception? A handler can trivially be made smart enough to notice it's being called over and over again in reaction to the death of one and the same process X. Actually, it does not need to be smart at all: when called, it will deal with the resp. communicator/file/win. It's the library that needs to be smart to mark process X failure only once and make sure all resp. communicators get notified in due time. > > Moreover, if inheritance is disabled in this case, we're more likely to get wrong programs. People will just keep forgetting about this special feature of the failure handlers, and fail in an uncontrolled fashion despite their best effort - a few weeks into the run. > > By excluding the inheritance here we create an exception to the rule that should be justified by more than our assumption of what the handler will do I think. > >>>> [Josh] If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. ï¿½If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. > > Again, this is not something I'd do. We may force the user to react to many process failures in one handler call. This is not what they may want to do. It's easier to write a handler that deals with one process failure at a time, and let the library care of invoking it in the relatively rare case of multiple process failures. Even if we think about multi- and manycore cases, the number of failed processes will normally be limited by the number of cores per node, which is not going to be astronomic for quite a while I guess. I.e., we don't seem to be creating a scalability bottleneck here. > > If we do care about scalability even here, we should rather give the user a way to tell the library: "Yeah, I've dealt with this dead process X you've just reported. By the way, I've also dealt with processes Y1 to Yn you haven't reported yet, but you'll understand, right?" Then, and only then may the library skip notifying the user of the process Y1 thru Yn failures on the same comm/file/win . It still has to report all failures on other comm/file/win, though. > > To simplify this, the library might report not one but a list of failures in the failure handler, or at least their total number for the moment of the handler invocation. Then the user will be well equipped to handle them all in one go. If he misses one, by chance or by sloppiness or by race condition, he'll get a shout (or shouts) next time. > > I'm not as deep into this proposal to judge at the moment if this case is already covered, so, take this as a user request. They will mostly be rather naive about MPI FT at first, but they will have healthy instincts, common sense, and some MPI experience for sure. The above treatise is a reflection on what I think they will expect of us on this backdrop. > >> [Josh] Thinking a bit about the implementation it should not be too bad to >> track such things. We could keep a boolean (or do some fun function >> pointer hacking) on the communicator that is flipped whenever a new >> failure is detected, then flip it back after firing the error handler. >> Similar boolean to what we might use to disable collectives. > > Looking above and summarizing, we probably need to track a process status first and once somewhere inside the implementation. When one process fails, all resp. communicators should be marked as potentially problematic, or another, more scalable mechanism should be used in reactive, on-demand fashion. Then the resp. calls on them should fire the resp. failure handlers at appropriate times (pt2pt op w/ the failed process, a collective op invocation, etc.) w/o any assumptions as to how much the user will want to do there. The users should in turn have a way to tell the library that they've dealt with more than just the immediately reported failure(s) that caused the handler to be invoked in the first place. > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey > Sent: Saturday, October 22, 2011 12:20 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] Latest version of chapter > > On Fri, Oct 21, 2011 at 6:01 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> >> On Oct 21, 2011, at 4:18 PM, Josh Hursey wrote: >> >>> The original problem was that the application wanted uniform >>> notification of a process failure (restricted to the set of processes >>> in the group associated with the communication objection). The current >>> error handlers are only fired when interacting with the failed process >>> directly (P2P) or indirectly (ANY_SOURCE, collectives). >>> >>> The requesters (who I believe are on the list and may want to pipe up) >>> were ok with having the callback triggered at an MPI boundary - so not >>> really asynchronous just not associated with the call. >>> >>> Maybe it is enough to restrict the notification to operations on the >>> communicator. So the FailHandler registered on commA is only fired >>> when commA is being used. The application would have to register the >>> FailHandler on all communicators that it is using and wants >>> notification from. But that would preserve some separation between the >>> library and application. >> >> I think that makes sense. > > Why don't we start with that restriction, and run it by folks next week. > >> >>> >>> A few things that probably should be clarified with the new FailHandler: >>> - Is it inherited by new communicators like other error handlers? >> >> I'd say no. ï¿½Because all it would do is call the same handler once for every communicator for the failure of the same process. > > I agree. > >> >>> - Without the communicator scope restriction mentioned above, if a >>> process fails, does it fire all of the FailHanders registered on >>> communication objects containing that process? (I think yes) If so, we >>> should probably state that we do not guarantee any ordering of these >>> calls. >> >> If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. ï¿½If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. > > > That sounds good to me. > > Thinking a bit about the implementation it should not be too bad to > track such things. We could keep a boolean (or do some fun function > pointer hacking) on the communicator that is flipped whenever a new > failure is detected, then flip it back after firing the error handler. > Similar boolean to what we might use to disable collectives. > >> >>> - In the function signatures the errhandler is 'int'/'integer' and >>> should probably be MPI_Errhandler or similar handle. >> >> I copied the prototypes from the error handler section. > > In section 8.3.1 of MPI 2.2 they are of the type MPI_Errhandler. So > pointers to the error handler function prototype. We can probably use > the same function pointer signature and object for these new > functions. > >> >>> - On the topic of what functions you can use inside, we can probably >>> use the language from the error handlers. I think it allows the user >>> to do pretty much anything they want, though I'd have to double check. >>> It might be that the standard is silent on this point, so no specific >>> restrictions are defined. >> >> I didn't see any restrictions, but then the standard says that all bets are off when you get an error, so calling anything at that point is undefined. > > I think staying silent for now is a good idea. But maybe we can think > about it over the weekend and talk more about it next week. > > > Darius: Do you have some time to make some of these changes to the > chapter and post a new copy of the document to the ticket? We probably > want to whole MPI standard text since some text changed outside of the > chapter for MPI_Finalize stuff. If not, I can probably get to it late > this evening, or tomorrow. > > > Thanks, > Josh > > >> >> -d >> >> >>> >>> >>> -- Josh >>> >>> >>> On Fri, Oct 21, 2011 at 4:54 PM, Supalov, Alexander >>> <alexander.supalov(a)intel.com> wrote: >>>> Imagine I use some data protection scheme inside B. I won't be affected by "wrong" libraries that I call before or after my protection is on. I may be affected by an asynchronous call out of "another world" that is possible if handlerA is called from within my library B. I.e., in the sequence >>>> >>>> A-B-A >>>> >>>> this extension allows B to be "hacked" by A by just killing one process at the right time. Moreover, I can clean up the callbacks by using MPI_Comm_create instead of MPI_Comm_dup. I cannot prevent an asynchronous handler from being called. >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>> Sent: Friday, October 21, 2011 10:44 PM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>> >>>> >>>> On Oct 21, 2011, at 3:19 PM, Supalov, Alexander wrote: >>>> >>>>> Thanks. See below (prefix "AS>"). >>>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>> Sent: Friday, October 21, 2011 9:57 PM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>> >>>>> >>>>> Let's say you have commA used by library A and commB used by library B. >>>>> Library A has registered the proc failure handler called handlerA on commA. >>>>> >>>>> Now, let's say a process that's in commA but not commB failed, and the thread is executing in library B and calls, e.g., ï¿½MPI_Send(..., commB). >>>>> >>>>> The MPI implementation performs the MPI_Send operation normally, then calls handlerA(commA, MPI_ERR_PROC_FAIL_STOP), and returns from MPI_Send normally. >>>>> >>>>> While in handlerA, the subject communicator (commA) is passed as a parameter, so it won't be out of scope. >>>>> >>>>> Is it a problem that library A's handler is called from "within" library B? >>>>> >>>>> AS> Sure. This handler may have been written by someone else who does not know me or my B or anything else. I may not even want it to be called from within my library B for security reasons. What if it unwinds the stack, connects to A's HQ, and dumps my confidential memory all over there? >>>> >>>> Yikes! ï¿½Don't link with libraries you don't trust :-) >>>> >>>> I don't know how to handle this case, but does the current standard prevent a library from snooping memory from other libraries? ï¿½A library could set an attribute with a copy callback function on comm_world. ï¿½That would be called from within another library's stack if that library tries to dup comm_world. >>>> >>>>> Moreover, by the time it's called, both A and commA may be the thing of the times long gone together with the context in which handlerA was supposed to be executed. What will it try to handle then and under what assumptions? I don't know. You? >>>> >>>> The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. >>>> >>>> -d >>>> >>>> >>>>> -d >>>>> >>>>> >>>>> On Oct 21, 2011, at 2:42 PM, Supalov, Alexander wrote: >>>>> >>>>>> Not really. How do you want the user make sense of that? E.g., I call A on commA, fail on commA asynchronously while calling a totally unrelated B on commB that has no failures in it, and am kicked out of B into someone else's error handler saying some "A" on "comma" failed? And what now? I may even have A and commA out of scope by then, possibly forever. >>>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>>> Sent: Friday, October 21, 2011 9:35 PM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>> >>>>>> With (regular) error handlers, they'll be called from within the function that raises the error. ï¿½With failure notification, because they're being called as a result of an external event (process failure), you could be called from within any function, even one not related to the comm/file/win that you registered the process failure notification handler on. >>>>>> >>>>>> Does that make sense? >>>>>> >>>>>> -d >>>>>> >>>>>> On Oct 21, 2011, at 1:51 PM, Sur, Sayantan wrote: >>>>>> >>>>>>> 17.5.1:11-12 - "The error handler function will be called by the MPI implementation from within the context of some MPI function that was called by the user." >>>>>>> >>>>>>> Maybe we should that error handlers are called from MPI functions that are associated with that comm/file/win? >>>>>>> >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>>>>>>> Sent: Friday, October 21, 2011 10:28 AM >>>>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>>>> >>>>>>>> I just wanted to note that we want to distribute a copy of this >>>>>>>> chapter to the MPI Forum before the meeting. As such we are planning >>>>>>>> on sending out a copy at COB today (so Friday ~5:00 pm EDT) so that >>>>>>>> people have an opportunity to look at the document before the Monday >>>>>>>> plenary. So please send any edits or comments before COB today, so we >>>>>>>> can work them into the draft. >>>>>>>> >>>>>>>> We will post the draft to the ticket, so that people know where to >>>>>>>> look for the current draft. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Josh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 20, 2011 at 7:06 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Here's the latest version of the FT chapter is on the wiki (it's in a >>>>>>>> new location on the main FT page under "ticket #276". ï¿½Please have a >>>>>>>> look and comment. >>>>>>>>> >>>>>>>>> Here's a direct link to the PDF: >>>>>>>>> ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/raw- >>>>>>>> attachment/wiki/FaultToleranceWikiPage/ft.pdf >>>>>>>>> >>>>>>>>> Here's a summary of the changes Josh and I made: >>>>>>>>> >>>>>>>>> * Minor wording touchups >>>>>>>>> * Added new semantic for MPI_ANY_SOURCE with the >>>>>>>> MPI_ERR_ANY_SOURCE_DISABLED error code >>>>>>>>> * Coverted wording for all comm, win, fh creation operations to not >>>>>>>> require collectively active communicators (eliminate requirement for >>>>>>>> synchronization) >>>>>>>>> * Added missing reader_lock to ANY_SOURCE example >>>>>>>>> * Added case for MPI_WIN_TEST >>>>>>>>> >>>>>>>>> and >>>>>>>>> >>>>>>>>> One-sided section >>>>>>>>> ï¿½clarified that window creation need not be blocking >>>>>>>>> ï¿½clarified that RMA ops might not complete correctly even if >>>>>>>>> ï¿½ ï¿½synchronization ops complete without error due to process >>>>>>>>> ï¿½ ï¿½failures >>>>>>>>> Process failure notification >>>>>>>>> ï¿½Added section describing new functions to add callbacks to comms, >>>>>>>>> ï¿½ ï¿½wins and files that are called when proc failure is detected >>>>>>>>> Other wordsmithing/cleanup changes >>>>>>>>> >>>>>>>>> -d >>>>>>>>> _______________________________________________ >>>>>>>>> mpi3-ft mailing list >>>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Joshua Hursey >>>>>>>> Postdoctoral Research Associate >>>>>>>> Oak Ridge National Laboratory >>>>>>>> http://users.nccs.gov/~jjhursey >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> -------------------------------------------------------------------------------------- >>>>>> Intel GmbH >>>>>> Dornacher Strasse 1 >>>>>> 85622 Feldkirchen/Muenchen, Deutschland >>>>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>>>> Registergericht: Muenchen HRB 47456 >>>>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> -------------------------------------------------------------------------------------- >>>>> Intel GmbH >>>>> Dornacher Strasse 1 >>>>> 85622 Feldkirchen/Muenchen, Deutschland >>>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>>> Registergericht: Muenchen HRB 47456 >>>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> -------------------------------------------------------------------------------------- >>>> Intel GmbH >>>> Dornacher Strasse 1 >>>> 85622 Feldkirchen/Muenchen, Deutschland >>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>> Registergericht: Muenchen HRB 47456 >>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>> >>> >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -------------------------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen, Deutschland > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 > Ust.-IdNr./VAT Registration No.: DE129385895 > Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Latest version of chapter
by Supalov, Alexander 21 Oct '11

21 Oct '11

PS. Small extension of the summary at the bottom of my message below, to cross all t's: "... should fire the resp. _inheritable_ failure handlers ..." -----Original Message----- From: Supalov, Alexander Sent: Saturday, October 22, 2011 6:01 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: RE: [Mpi3-ft] Latest version of chapter Hi, I think what emerges is an approach to a good solution. There are some loose ends, though. Going thru the discussion: >>>> [Darius] The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. Let's clarify this. Imagine I had commA with some process X in it. X failed when commA was being destroyed. I did not get the notification in the resp. call on my process Y because it had already returned there. Later I, being totally unaware of the process X failure, try to create a new commB with process X in it. Will I notice that it's gone? How will I detect that now technically as an MPI implementor? Probably, I'll track process state deep inside the library. How will I detect that as an MPI user - when my handlerA is gone? Probably, I want handlerB, if available, to be called at this moment. I hope this example is reasonable as it partially underpins the remaining comments below. >>> [Josh] A few things that probably should be clarified with the new FailHandler: >>> - Is it inherited by new communicators like other error handlers? >> >> [Darius] I'd say no. Because all it would do is call the same handler once for every communicator for the failure of the same process. > >[Josh] I agree. Why this exception? A handler can trivially be made smart enough to notice it's being called over and over again in reaction to the death of one and the same process X. Actually, it does not need to be smart at all: when called, it will deal with the resp. communicator/file/win. It's the library that needs to be smart to mark process X failure only once and make sure all resp. communicators get notified in due time. Moreover, if inheritance is disabled in this case, we're more likely to get wrong programs. People will just keep forgetting about this special feature of the failure handlers, and fail in an uncontrolled fashion despite their best effort - a few weeks into the run. By excluding the inheritance here we create an exception to the rule that should be justified by more than our assumption of what the handler will do I think. >>> [Josh] If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. Again, this is not something I'd do. We may force the user to react to many process failures in one handler call. This is not what they may want to do. It's easier to write a handler that deals with one process failure at a time, and let the library care of invoking it in the relatively rare case of multiple process failures. Even if we think about multi- and manycore cases, the number of failed processes will normally be limited by the number of cores per node, which is not going to be astronomic for quite a while I guess. I.e., we don't seem to be creating a scalability bottleneck here. If we do care about scalability even here, we should rather give the user a way to tell the library: "Yeah, I've dealt with this dead process X you've just reported. By the way, I've also dealt with processes Y1 to Yn you haven't reported yet, but you'll understand, right?" Then, and only then may the library skip notifying the user of the process Y1 thru Yn failures on the same comm/file/win . It still has to report all failures on other comm/file/win, though. To simplify this, the library might report not one but a list of failures in the failure handler, or at least their total number for the moment of the handler invocation. Then the user will be well equipped to handle them all in one go. If he misses one, by chance or by sloppiness or by race condition, he'll get a shout (or shouts) next time. I'm not as deep into this proposal to judge at the moment if this case is already covered, so, take this as a user request. They will mostly be rather naive about MPI FT at first, but they will have healthy instincts, common sense, and some MPI experience for sure. The above treatise is a reflection on what I think they will expect of us on this backdrop. > [Josh] Thinking a bit about the implementation it should not be too bad to > track such things. We could keep a boolean (or do some fun function > pointer hacking) on the communicator that is flipped whenever a new > failure is detected, then flip it back after firing the error handler. > Similar boolean to what we might use to disable collectives. Looking above and summarizing, we probably need to track a process status first and once somewhere inside the implementation. When one process fails, all resp. communicators should be marked as potentially problematic, or another, more scalable mechanism should be used in reactive, on-demand fashion. Then the resp. calls on them should fire the resp. failure handlers at appropriate times (pt2pt op w/ the failed process, a collective op invocation, etc.) w/o any assumptions as to how much the user will want to do there. The users should in turn have a way to tell the library that they've dealt with more than just the immediately reported failure(s) that caused the handler to be invoked in the first place. Best regards. Alexander -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Saturday, October 22, 2011 12:20 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Latest version of chapter On Fri, Oct 21, 2011 at 6:01 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > On Oct 21, 2011, at 4:18 PM, Josh Hursey wrote: > >> The original problem was that the application wanted uniform >> notification of a process failure (restricted to the set of processes >> in the group associated with the communication objection). The current >> error handlers are only fired when interacting with the failed process >> directly (P2P) or indirectly (ANY_SOURCE, collectives). >> >> The requesters (who I believe are on the list and may want to pipe up) >> were ok with having the callback triggered at an MPI boundary - so not >> really asynchronous just not associated with the call. >> >> Maybe it is enough to restrict the notification to operations on the >> communicator. So the FailHandler registered on commA is only fired >> when commA is being used. The application would have to register the >> FailHandler on all communicators that it is using and wants >> notification from. But that would preserve some separation between the >> library and application. > > I think that makes sense. Why don't we start with that restriction, and run it by folks next week. > >> >> A few things that probably should be clarified with the new FailHandler: >> - Is it inherited by new communicators like other error handlers? > > I'd say no. Because all it would do is call the same handler once for every communicator for the failure of the same process. I agree. > >> - Without the communicator scope restriction mentioned above, if a >> process fails, does it fire all of the FailHanders registered on >> communication objects containing that process? (I think yes) If so, we >> should probably state that we do not guarantee any ordering of these >> calls. > > If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. That sounds good to me. Thinking a bit about the implementation it should not be too bad to track such things. We could keep a boolean (or do some fun function pointer hacking) on the communicator that is flipped whenever a new failure is detected, then flip it back after firing the error handler. Similar boolean to what we might use to disable collectives. > >> - In the function signatures the errhandler is 'int'/'integer' and >> should probably be MPI_Errhandler or similar handle. > > I copied the prototypes from the error handler section. In section 8.3.1 of MPI 2.2 they are of the type MPI_Errhandler. So pointers to the error handler function prototype. We can probably use the same function pointer signature and object for these new functions. > >> - On the topic of what functions you can use inside, we can probably >> use the language from the error handlers. I think it allows the user >> to do pretty much anything they want, though I'd have to double check. >> It might be that the standard is silent on this point, so no specific >> restrictions are defined. > > I didn't see any restrictions, but then the standard says that all bets are off when you get an error, so calling anything at that point is undefined. I think staying silent for now is a good idea. But maybe we can think about it over the weekend and talk more about it next week. Darius: Do you have some time to make some of these changes to the chapter and post a new copy of the document to the ticket? We probably want to whole MPI standard text since some text changed outside of the chapter for MPI_Finalize stuff. If not, I can probably get to it late this evening, or tomorrow. Thanks, Josh > > -d > > >> >> >> -- Josh >> >> >> On Fri, Oct 21, 2011 at 4:54 PM, Supalov, Alexander >> <alexander.supalov(a)intel.com> wrote: >>> Imagine I use some data protection scheme inside B. I won't be affected by "wrong" libraries that I call before or after my protection is on. I may be affected by an asynchronous call out of "another world" that is possible if handlerA is called from within my library B. I.e., in the sequence >>> >>> A-B-A >>> >>> this extension allows B to be "hacked" by A by just killing one process at the right time. Moreover, I can clean up the callbacks by using MPI_Comm_create instead of MPI_Comm_dup. I cannot prevent an asynchronous handler from being called. >>> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>> Sent: Friday, October 21, 2011 10:44 PM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> Subject: Re: [Mpi3-ft] Latest version of chapter >>> >>> >>> On Oct 21, 2011, at 3:19 PM, Supalov, Alexander wrote: >>> >>>> Thanks. See below (prefix "AS>"). >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>> Sent: Friday, October 21, 2011 9:57 PM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>> >>>> >>>> Let's say you have commA used by library A and commB used by library B. >>>> Library A has registered the proc failure handler called handlerA on commA. >>>> >>>> Now, let's say a process that's in commA but not commB failed, and the thread is executing in library B and calls, e.g., MPI_Send(..., commB). >>>> >>>> The MPI implementation performs the MPI_Send operation normally, then calls handlerA(commA, MPI_ERR_PROC_FAIL_STOP), and returns from MPI_Send normally. >>>> >>>> While in handlerA, the subject communicator (commA) is passed as a parameter, so it won't be out of scope. >>>> >>>> Is it a problem that library A's handler is called from "within" library B? >>>> >>>> AS> Sure. This handler may have been written by someone else who does not know me or my B or anything else. I may not even want it to be called from within my library B for security reasons. What if it unwinds the stack, connects to A's HQ, and dumps my confidential memory all over there? >>> >>> Yikes! Don't link with libraries you don't trust :-) >>> >>> I don't know how to handle this case, but does the current standard prevent a library from snooping memory from other libraries? A library could set an attribute with a copy callback function on comm_world. That would be called from within another library's stack if that library tries to dup comm_world. >>> >>>> Moreover, by the time it's called, both A and commA may be the thing of the times long gone together with the context in which handlerA was supposed to be executed. What will it try to handle then and under what assumptions? I don't know. You? >>> >>> The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. >>> >>> -d >>> >>> >>>> -d >>>> >>>> >>>> On Oct 21, 2011, at 2:42 PM, Supalov, Alexander wrote: >>>> >>>>> Not really. How do you want the user make sense of that? E.g., I call A on commA, fail on commA asynchronously while calling a totally unrelated B on commB that has no failures in it, and am kicked out of B into someone else's error handler saying some "A" on "comma" failed? And what now? I may even have A and commA out of scope by then, possibly forever. >>>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>> Sent: Friday, October 21, 2011 9:35 PM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>> >>>>> With (regular) error handlers, they'll be called from within the function that raises the error. With failure notification, because they're being called as a result of an external event (process failure), you could be called from within any function, even one not related to the comm/file/win that you registered the process failure notification handler on. >>>>> >>>>> Does that make sense? >>>>> >>>>> -d >>>>> >>>>> On Oct 21, 2011, at 1:51 PM, Sur, Sayantan wrote: >>>>> >>>>>> 17.5.1:11-12 - "The error handler function will be called by the MPI implementation from within the context of some MPI function that was called by the user." >>>>>> >>>>>> Maybe we should that error handlers are called from MPI functions that are associated with that comm/file/win? >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>>>>>> Sent: Friday, October 21, 2011 10:28 AM >>>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>>> >>>>>>> I just wanted to note that we want to distribute a copy of this >>>>>>> chapter to the MPI Forum before the meeting. As such we are planning >>>>>>> on sending out a copy at COB today (so Friday ~5:00 pm EDT) so that >>>>>>> people have an opportunity to look at the document before the Monday >>>>>>> plenary. So please send any edits or comments before COB today, so we >>>>>>> can work them into the draft. >>>>>>> >>>>>>> We will post the draft to the ticket, so that people know where to >>>>>>> look for the current draft. >>>>>>> >>>>>>> Thanks, >>>>>>> Josh >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 20, 2011 at 7:06 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> >>>>>>> wrote: >>>>>>>> >>>>>>>> Here's the latest version of the FT chapter is on the wiki (it's in a >>>>>>> new location on the main FT page under "ticket #276". Please have a >>>>>>> look and comment. >>>>>>>> >>>>>>>> Here's a direct link to the PDF: >>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/raw- >>>>>>> attachment/wiki/FaultToleranceWikiPage/ft.pdf >>>>>>>> >>>>>>>> Here's a summary of the changes Josh and I made: >>>>>>>> >>>>>>>> * Minor wording touchups >>>>>>>> * Added new semantic for MPI_ANY_SOURCE with the >>>>>>> MPI_ERR_ANY_SOURCE_DISABLED error code >>>>>>>> * Coverted wording for all comm, win, fh creation operations to not >>>>>>> require collectively active communicators (eliminate requirement for >>>>>>> synchronization) >>>>>>>> * Added missing reader_lock to ANY_SOURCE example >>>>>>>> * Added case for MPI_WIN_TEST >>>>>>>> >>>>>>>> and >>>>>>>> >>>>>>>> One-sided section >>>>>>>> clarified that window creation need not be blocking >>>>>>>> clarified that RMA ops might not complete correctly even if >>>>>>>> synchronization ops complete without error due to process >>>>>>>> failures >>>>>>>> Process failure notification >>>>>>>> Added section describing new functions to add callbacks to comms, >>>>>>>> wins and files that are called when proc failure is detected >>>>>>>> Other wordsmithing/cleanup changes >>>>>>>> >>>>>>>> -d >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Joshua Hursey >>>>>>> Postdoctoral Research Associate >>>>>>> Oak Ridge National Laboratory >>>>>>> http://users.nccs.gov/~jjhursey >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> -------------------------------------------------------------------------------------- >>>>> Intel GmbH >>>>> Dornacher Strasse 1 >>>>> 85622 Feldkirchen/Muenchen, Deutschland >>>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>>> Registergericht: Muenchen HRB 47456 >>>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> -------------------------------------------------------------------------------------- >>>> Intel GmbH >>>> Dornacher Strasse 1 >>>> 85622 Feldkirchen/Muenchen, Deutschland >>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>> Registergericht: Muenchen HRB 47456 >>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> -------------------------------------------------------------------------------------- >>> Intel GmbH >>> Dornacher Strasse 1 >>> 85622 Feldkirchen/Muenchen, Deutschland >>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>> Registergericht: Muenchen HRB 47456 >>> Ust.-IdNr./VAT Registration No.: DE129385895 >>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi3-ft] Latest version of chapter
by Supalov, Alexander 21 Oct '11

21 Oct '11

Hi, I think what emerges is an approach to a good solution. There are some loose ends, though. Going thru the discussion: >>>> [Darius] The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. Let's clarify this. Imagine I had commA with some process X in it. X failed when commA was being destroyed. I did not get the notification in the resp. call on my process Y because it had already returned there. Later I, being totally unaware of the process X failure, try to create a new commB with process X in it. Will I notice that it's gone? How will I detect that now technically as an MPI implementor? Probably, I'll track process state deep inside the library. How will I detect that as an MPI user - when my handlerA is gone? Probably, I want handlerB, if available, to be called at this moment. I hope this example is reasonable as it partially underpins the remaining comments below. >>> [Josh] A few things that probably should be clarified with the new FailHandler: >>> - Is it inherited by new communicators like other error handlers? >> >> [Darius] I'd say no. Because all it would do is call the same handler once for every communicator for the failure of the same process. > >[Josh] I agree. Why this exception? A handler can trivially be made smart enough to notice it's being called over and over again in reaction to the death of one and the same process X. Actually, it does not need to be smart at all: when called, it will deal with the resp. communicator/file/win. It's the library that needs to be smart to mark process X failure only once and make sure all resp. communicators get notified in due time. Moreover, if inheritance is disabled in this case, we're more likely to get wrong programs. People will just keep forgetting about this special feature of the failure handlers, and fail in an uncontrolled fashion despite their best effort - a few weeks into the run. By excluding the inheritance here we create an exception to the rule that should be justified by more than our assumption of what the handler will do I think. >>> [Josh] If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. Again, this is not something I'd do. We may force the user to react to many process failures in one handler call. This is not what they may want to do. It's easier to write a handler that deals with one process failure at a time, and let the library care of invoking it in the relatively rare case of multiple process failures. Even if we think about multi- and manycore cases, the number of failed processes will normally be limited by the number of cores per node, which is not going to be astronomic for quite a while I guess. I.e., we don't seem to be creating a scalability bottleneck here. If we do care about scalability even here, we should rather give the user a way to tell the library: "Yeah, I've dealt with this dead process X you've just reported. By the way, I've also dealt with processes Y1 to Yn you haven't reported yet, but you'll understand, right?" Then, and only then may the library skip notifying the user of the process Y1 thru Yn failures on the same comm/file/win . It still has to report all failures on other comm/file/win, though. To simplify this, the library might report not one but a list of failures in the failure handler, or at least their total number for the moment of the handler invocation. Then the user will be well equipped to handle them all in one go. If he misses one, by chance or by sloppiness or by race condition, he'll get a shout (or shouts) next time. I'm not as deep into this proposal to judge at the moment if this case is already covered, so, take this as a user request. They will mostly be rather naive about MPI FT at first, but they will have healthy instincts, common sense, and some MPI experience for sure. The above treatise is a reflection on what I think they will expect of us on this backdrop. > [Josh] Thinking a bit about the implementation it should not be too bad to > track such things. We could keep a boolean (or do some fun function > pointer hacking) on the communicator that is flipped whenever a new > failure is detected, then flip it back after firing the error handler. > Similar boolean to what we might use to disable collectives. Looking above and summarizing, we probably need to track a process status first and once somewhere inside the implementation. When one process fails, all resp. communicators should be marked as potentially problematic, or another, more scalable mechanism should be used in reactive, on-demand fashion. Then the resp. calls on them should fire the resp. failure handlers at appropriate times (pt2pt op w/ the failed process, a collective op invocation, etc.) w/o any assumptions as to how much the user will want to do there. The users should in turn have a way to tell the library that they've dealt with more than just the immediately reported failure(s) that caused the handler to be invoked in the first place. Best regards. Alexander -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Saturday, October 22, 2011 12:20 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Latest version of chapter On Fri, Oct 21, 2011 at 6:01 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > On Oct 21, 2011, at 4:18 PM, Josh Hursey wrote: > >> The original problem was that the application wanted uniform >> notification of a process failure (restricted to the set of processes >> in the group associated with the communication objection). The current >> error handlers are only fired when interacting with the failed process >> directly (P2P) or indirectly (ANY_SOURCE, collectives). >> >> The requesters (who I believe are on the list and may want to pipe up) >> were ok with having the callback triggered at an MPI boundary - so not >> really asynchronous just not associated with the call. >> >> Maybe it is enough to restrict the notification to operations on the >> communicator. So the FailHandler registered on commA is only fired >> when commA is being used. The application would have to register the >> FailHandler on all communicators that it is using and wants >> notification from. But that would preserve some separation between the >> library and application. > > I think that makes sense. Why don't we start with that restriction, and run it by folks next week. > >> >> A few things that probably should be clarified with the new FailHandler: >> - Is it inherited by new communicators like other error handlers? > > I'd say no. Because all it would do is call the same handler once for every communicator for the failure of the same process. I agree. > >> - Without the communicator scope restriction mentioned above, if a >> process fails, does it fire all of the FailHanders registered on >> communication objects containing that process? (I think yes) If so, we >> should probably state that we do not guarantee any ordering of these >> calls. > > If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. That sounds good to me. Thinking a bit about the implementation it should not be too bad to track such things. We could keep a boolean (or do some fun function pointer hacking) on the communicator that is flipped whenever a new failure is detected, then flip it back after firing the error handler. Similar boolean to what we might use to disable collectives. > >> - In the function signatures the errhandler is 'int'/'integer' and >> should probably be MPI_Errhandler or similar handle. > > I copied the prototypes from the error handler section. In section 8.3.1 of MPI 2.2 they are of the type MPI_Errhandler. So pointers to the error handler function prototype. We can probably use the same function pointer signature and object for these new functions. > >> - On the topic of what functions you can use inside, we can probably >> use the language from the error handlers. I think it allows the user >> to do pretty much anything they want, though I'd have to double check. >> It might be that the standard is silent on this point, so no specific >> restrictions are defined. > > I didn't see any restrictions, but then the standard says that all bets are off when you get an error, so calling anything at that point is undefined. I think staying silent for now is a good idea. But maybe we can think about it over the weekend and talk more about it next week. Darius: Do you have some time to make some of these changes to the chapter and post a new copy of the document to the ticket? We probably want to whole MPI standard text since some text changed outside of the chapter for MPI_Finalize stuff. If not, I can probably get to it late this evening, or tomorrow. Thanks, Josh > > -d > > >> >> >> -- Josh >> >> >> On Fri, Oct 21, 2011 at 4:54 PM, Supalov, Alexander >> <alexander.supalov(a)intel.com> wrote: >>> Imagine I use some data protection scheme inside B. I won't be affected by "wrong" libraries that I call before or after my protection is on. I may be affected by an asynchronous call out of "another world" that is possible if handlerA is called from within my library B. I.e., in the sequence >>> >>> A-B-A >>> >>> this extension allows B to be "hacked" by A by just killing one process at the right time. Moreover, I can clean up the callbacks by using MPI_Comm_create instead of MPI_Comm_dup. I cannot prevent an asynchronous handler from being called. >>> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>> Sent: Friday, October 21, 2011 10:44 PM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> Subject: Re: [Mpi3-ft] Latest version of chapter >>> >>> >>> On Oct 21, 2011, at 3:19 PM, Supalov, Alexander wrote: >>> >>>> Thanks. See below (prefix "AS>"). >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>> Sent: Friday, October 21, 2011 9:57 PM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>> >>>> >>>> Let's say you have commA used by library A and commB used by library B. >>>> Library A has registered the proc failure handler called handlerA on commA. >>>> >>>> Now, let's say a process that's in commA but not commB failed, and the thread is executing in library B and calls, e.g., MPI_Send(..., commB). >>>> >>>> The MPI implementation performs the MPI_Send operation normally, then calls handlerA(commA, MPI_ERR_PROC_FAIL_STOP), and returns from MPI_Send normally. >>>> >>>> While in handlerA, the subject communicator (commA) is passed as a parameter, so it won't be out of scope. >>>> >>>> Is it a problem that library A's handler is called from "within" library B? >>>> >>>> AS> Sure. This handler may have been written by someone else who does not know me or my B or anything else. I may not even want it to be called from within my library B for security reasons. What if it unwinds the stack, connects to A's HQ, and dumps my confidential memory all over there? >>> >>> Yikes! Don't link with libraries you don't trust :-) >>> >>> I don't know how to handle this case, but does the current standard prevent a library from snooping memory from other libraries? A library could set an attribute with a copy callback function on comm_world. That would be called from within another library's stack if that library tries to dup comm_world. >>> >>>> Moreover, by the time it's called, both A and commA may be the thing of the times long gone together with the context in which handlerA was supposed to be executed. What will it try to handle then and under what assumptions? I don't know. You? >>> >>> The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. >>> >>> -d >>> >>> >>>> -d >>>> >>>> >>>> On Oct 21, 2011, at 2:42 PM, Supalov, Alexander wrote: >>>> >>>>> Not really. How do you want the user make sense of that? E.g., I call A on commA, fail on commA asynchronously while calling a totally unrelated B on commB that has no failures in it, and am kicked out of B into someone else's error handler saying some "A" on "comma" failed? And what now? I may even have A and commA out of scope by then, possibly forever. >>>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>> Sent: Friday, October 21, 2011 9:35 PM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>> >>>>> With (regular) error handlers, they'll be called from within the function that raises the error. With failure notification, because they're being called as a result of an external event (process failure), you could be called from within any function, even one not related to the comm/file/win that you registered the process failure notification handler on. >>>>> >>>>> Does that make sense? >>>>> >>>>> -d >>>>> >>>>> On Oct 21, 2011, at 1:51 PM, Sur, Sayantan wrote: >>>>> >>>>>> 17.5.1:11-12 - "The error handler function will be called by the MPI implementation from within the context of some MPI function that was called by the user." >>>>>> >>>>>> Maybe we should that error handlers are called from MPI functions that are associated with that comm/file/win? >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>>>>>> Sent: Friday, October 21, 2011 10:28 AM >>>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>>> >>>>>>> I just wanted to note that we want to distribute a copy of this >>>>>>> chapter to the MPI Forum before the meeting. As such we are planning >>>>>>> on sending out a copy at COB today (so Friday ~5:00 pm EDT) so that >>>>>>> people have an opportunity to look at the document before the Monday >>>>>>> plenary. So please send any edits or comments before COB today, so we >>>>>>> can work them into the draft. >>>>>>> >>>>>>> We will post the draft to the ticket, so that people know where to >>>>>>> look for the current draft. >>>>>>> >>>>>>> Thanks, >>>>>>> Josh >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 20, 2011 at 7:06 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> >>>>>>> wrote: >>>>>>>> >>>>>>>> Here's the latest version of the FT chapter is on the wiki (it's in a >>>>>>> new location on the main FT page under "ticket #276". Please have a >>>>>>> look and comment. >>>>>>>> >>>>>>>> Here's a direct link to the PDF: >>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/raw- >>>>>>> attachment/wiki/FaultToleranceWikiPage/ft.pdf >>>>>>>> >>>>>>>> Here's a summary of the changes Josh and I made: >>>>>>>> >>>>>>>> * Minor wording touchups >>>>>>>> * Added new semantic for MPI_ANY_SOURCE with the >>>>>>> MPI_ERR_ANY_SOURCE_DISABLED error code >>>>>>>> * Coverted wording for all comm, win, fh creation operations to not >>>>>>> require collectively active communicators (eliminate requirement for >>>>>>> synchronization) >>>>>>>> * Added missing reader_lock to ANY_SOURCE example >>>>>>>> * Added case for MPI_WIN_TEST >>>>>>>> >>>>>>>> and >>>>>>>> >>>>>>>> One-sided section >>>>>>>> clarified that window creation need not be blocking >>>>>>>> clarified that RMA ops might not complete correctly even if >>>>>>>> synchronization ops complete without error due to process >>>>>>>> failures >>>>>>>> Process failure notification >>>>>>>> Added section describing new functions to add callbacks to comms, >>>>>>>> wins and files that are called when proc failure is detected >>>>>>>> Other wordsmithing/cleanup changes >>>>>>>> >>>>>>>> -d >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Joshua Hursey >>>>>>> Postdoctoral Research Associate >>>>>>> Oak Ridge National Laboratory >>>>>>> http://users.nccs.gov/~jjhursey >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> -------------------------------------------------------------------------------------- >>>>> Intel GmbH >>>>> Dornacher Strasse 1 >>>>> 85622 Feldkirchen/Muenchen, Deutschland >>>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>>> Registergericht: Muenchen HRB 47456 >>>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> -------------------------------------------------------------------------------------- >>>> Intel GmbH >>>> Dornacher Strasse 1 >>>> 85622 Feldkirchen/Muenchen, Deutschland >>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>> Registergericht: Muenchen HRB 47456 >>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> -------------------------------------------------------------------------------------- >>> Intel GmbH >>> Dornacher Strasse 1 >>> 85622 Feldkirchen/Muenchen, Deutschland >>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>> Registergericht: Muenchen HRB 47456 >>> Ust.-IdNr./VAT Registration No.: DE129385895 >>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi3-ft] one sided
by Josh Hursey 21 Oct '11

21 Oct '11

As an aside: We will probably be talking more about the communication objection creation semantics a bit more next week along with the one-sided FT semantics in general. -- Josh On Fri, Oct 21, 2011 at 6:08 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: > Thanks for the clarification, Josh. It makes sense to me. > >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >> Sent: Friday, October 21, 2011 2:28 PM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: Re: [Mpi3-ft] one sided >> >> Since MPI is creating a new communication object, we require that all >> processes have access to the object if it was created anywhere. >> >> So consider if the 'win' object was created at some processes and not >> others. The application with the valid 'win' calls MPI_Win_fence(). >> Other processes in the group associated with the window will not be to >> call MPI_Win_fence since they do not have a valid object. The program >> is erroneous since not all processes are calling the collective. So >> the semantics become muddled when we talk about collective operations >> (even like MPI_Win_free) when not all processes are guaranteed to have >> a valid communication object to use. >> >> So we just need the requirement that the object is either created >> everywhere or nowhere. Since we can only make statements about the >> behavior of MPI after the MPI_ERR_PROC_FAIL_STOP error code, we >> restrict the language to just that error code. Though it could be >> argued that this is a more general requirement, but that is slightly >> out of scope for this proposal. >> >> Does that help clarify? >> >> -- Josh >> >> On Fri, Oct 21, 2011 at 4:54 PM, Sur, Sayantan <sayantan.sur(a)intel.com> >> wrote: >> > Hi All, >> > >> > The new chapter says this about the window creation: >> > >> > "If the MPI_WIN_CREATE operation fails at any live process due to a >> process failure, then the operation must fail at every live process >> with an error in the class MPI_ERR_PROC_FAIL_STOP." >> > >> > I'm wondering what would happen if MPI_WIN_CREATE did not have this >> qualification at all. i.e. it would succeed at some processes and fail >> at some processes. After all, any following GET or PUT calls can always >> raise the error class MPI_ERR_PROC_FAIL_STOP. Also, the communicator >> passed to MPI_WIN_CREATE is allowed to have dead processes in it ... >> then why qualify win create with this requirement? >> > >> > Thanks, >> > Sayantan. >> > >> >> -----Original Message----- >> >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> >> bounces(a)lists.mpi-forum.org] On Behalf Of Pavan Balaji >> >> Sent: Wednesday, October 19, 2011 9:05 AM >> >> To: mpi3-ft(a)lists.mpi-forum.org >> >> Subject: Re: [Mpi3-ft] one sided >> >> >> >> >> >> FYI, you cannot "require" some behavior from MPI through an info >> >> argument. It is perfectly legitimate for the MPI implementation to >> >> completely ignore any info arguments passed. They are just user >> hints. >> >> >> >> ï¿½ -- Pavan >> >> >> >> On 10/19/2011 07:59 AM, Josh Hursey wrote: >> >> > Let's be sure to talk about this on today's call. I have some >> other >> >> > one-sided notes that I would like to go over as well. >> >> > >> >> > It would be fairly easy to support both modes since the >> >> MPI_Win_create >> >> > operation takes an info argument. We could define a key (similar >> to >> >> > what they have done for other operations) that either loosens or >> >> > tightens the semantics depending on what the default behavior >> should >> >> > be. >> >> > >> >> > I think it is ok to have a non-synchronizing option, just as long >> as >> >> > we have clear semantics for when the window is not created at all >> >> > processes due to some process failure - or if the window is always >> >> > created regardless of emerging failure then we might avoid this >> >> issue, >> >> > but that might require some additional clarification. >> >> > >> >> > Thanks, >> >> > Josh >> >> > >> >> > On Wed, Oct 19, 2011 at 4:11 AM, Supalov, Alexander >> >> > <alexander.supalov(a)intel.com> ï¿½wrote: >> >> >> Thanks. Why not having two calls or modes of operation to cover >> >> both? >> >> >> >> >> >> -----Original Message----- >> >> >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> >> bounces(a)lists.mpi-forum.org] On Behalf Of Darius Buntinas >> >> >> Sent: Tuesday, October 18, 2011 9:57 PM >> >> >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >> >> Group >> >> >> Subject: [Mpi3-ft] one sided >> >> >> >> >> >> >> >> >> I got some feedback from Jim and Pavan on the one-sided section. >> >> One thing Jim pointed out was that we don't want to make window >> >> creation synchronizing, and the fail-or-succeed everywhere >> requirement >> >> would do that. >> >> >> >> >> >> If we say that window creation should not fail due to failed >> >> processes, that would accomplish the same thing: ï¿½If a window is >> >> created by a correct program, then it will succeed at all live >> >> processes. ï¿½Note that if an incorrect program specifies invalid >> >> parameters then the window creation may fail at some processes and >> >> succeed at others, but this is what we already have today. >> >> >> >> >> >> However, it's possible that some implementations cannot satisfy >> this >> >> requirement because, e.g., they do collectives as part of the >> >> operation. ï¿½So maybe we should have two options: >> >> >> >> >> >> ï¿½ Either: >> >> >> ï¿½ ï¿½ window creation won't fail because if failed processes >> >> >> ï¿½ or >> >> >> ï¿½ ï¿½ window creation will either succeed or fail everywhere and if >> >> window creation fails at >> >> >> ï¿½ ï¿½ any process it fails at every process >> >> >> >> >> >> -d >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> mpi3-ft mailing list >> >> >> mpi3-ft(a)lists.mpi-forum.org >> >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> ----------------------------------------------------------------- >> --- >> >> ------------------ >> >> >> Intel GmbH >> >> >> Dornacher Strasse 1 >> >> >> 85622 Feldkirchen/Muenchen, Deutschland >> >> >> Sitz der Gesellschaft: Feldkirchen bei Muenchen >> >> >> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes >> Schwaderer >> >> >> Registergericht: Muenchen HRB 47456 >> >> >> Ust.-IdNr./VAT Registration No.: DE129385895 >> >> >> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> mpi3-ft mailing list >> >> >> mpi3-ft(a)lists.mpi-forum.org >> >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> >> >> >> >> >> > >> >> > >> >> > >> >> >> >> -- >> >> Pavan Balaji >> >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > >> > _______________________________________________ >> > mpi3-ft mailing list >> > mpi3-ft(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > >> > >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Latest version of chapter
by Josh Hursey 21 Oct '11

21 Oct '11

On Fri, Oct 21, 2011 at 6:01 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > On Oct 21, 2011, at 4:18 PM, Josh Hursey wrote: > >> The original problem was that the application wanted uniform >> notification of a process failure (restricted to the set of processes >> in the group associated with the communication objection). The current >> error handlers are only fired when interacting with the failed process >> directly (P2P) or indirectly (ANY_SOURCE, collectives). >> >> The requesters (who I believe are on the list and may want to pipe up) >> were ok with having the callback triggered at an MPI boundary - so not >> really asynchronous just not associated with the call. >> >> Maybe it is enough to restrict the notification to operations on the >> communicator. So the FailHandler registered on commA is only fired >> when commA is being used. The application would have to register the >> FailHandler on all communicators that it is using and wants >> notification from. But that would preserve some separation between the >> library and application. > > I think that makes sense. Why don't we start with that restriction, and run it by folks next week. > >> >> A few things that probably should be clarified with the new FailHandler: >> - Is it inherited by new communicators like other error handlers? > > I'd say no. ï¿½Because all it would do is call the same handler once for every communicator for the failure of the same process. I agree. > >> - Without the communicator scope restriction mentioned above, if a >> process fails, does it fire all of the FailHanders registered on >> communication objects containing that process? (I think yes) If so, we >> should probably state that we do not guarantee any ordering of these >> calls. > > If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. ï¿½If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. That sounds good to me. Thinking a bit about the implementation it should not be too bad to track such things. We could keep a boolean (or do some fun function pointer hacking) on the communicator that is flipped whenever a new failure is detected, then flip it back after firing the error handler. Similar boolean to what we might use to disable collectives. > >> - In the function signatures the errhandler is 'int'/'integer' and >> should probably be MPI_Errhandler or similar handle. > > I copied the prototypes from the error handler section. In section 8.3.1 of MPI 2.2 they are of the type MPI_Errhandler. So pointers to the error handler function prototype. We can probably use the same function pointer signature and object for these new functions. > >> - On the topic of what functions you can use inside, we can probably >> use the language from the error handlers. I think it allows the user >> to do pretty much anything they want, though I'd have to double check. >> It might be that the standard is silent on this point, so no specific >> restrictions are defined. > > I didn't see any restrictions, but then the standard says that all bets are off when you get an error, so calling anything at that point is undefined. I think staying silent for now is a good idea. But maybe we can think about it over the weekend and talk more about it next week. Darius: Do you have some time to make some of these changes to the chapter and post a new copy of the document to the ticket? We probably want to whole MPI standard text since some text changed outside of the chapter for MPI_Finalize stuff. If not, I can probably get to it late this evening, or tomorrow. Thanks, Josh > > -d > > >> >> >> -- Josh >> >> >> On Fri, Oct 21, 2011 at 4:54 PM, Supalov, Alexander >> <alexander.supalov(a)intel.com> wrote: >>> Imagine I use some data protection scheme inside B. I won't be affected by "wrong" libraries that I call before or after my protection is on. I may be affected by an asynchronous call out of "another world" that is possible if handlerA is called from within my library B. I.e., in the sequence >>> >>> A-B-A >>> >>> this extension allows B to be "hacked" by A by just killing one process at the right time. Moreover, I can clean up the callbacks by using MPI_Comm_create instead of MPI_Comm_dup. I cannot prevent an asynchronous handler from being called. >>> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>> Sent: Friday, October 21, 2011 10:44 PM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> Subject: Re: [Mpi3-ft] Latest version of chapter >>> >>> >>> On Oct 21, 2011, at 3:19 PM, Supalov, Alexander wrote: >>> >>>> Thanks. See below (prefix "AS>"). >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>> Sent: Friday, October 21, 2011 9:57 PM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>> >>>> >>>> Let's say you have commA used by library A and commB used by library B. >>>> Library A has registered the proc failure handler called handlerA on commA. >>>> >>>> Now, let's say a process that's in commA but not commB failed, and the thread is executing in library B and calls, e.g., ï¿½MPI_Send(..., commB). >>>> >>>> The MPI implementation performs the MPI_Send operation normally, then calls handlerA(commA, MPI_ERR_PROC_FAIL_STOP), and returns from MPI_Send normally. >>>> >>>> While in handlerA, the subject communicator (commA) is passed as a parameter, so it won't be out of scope. >>>> >>>> Is it a problem that library A's handler is called from "within" library B? >>>> >>>> AS> Sure. This handler may have been written by someone else who does not know me or my B or anything else. I may not even want it to be called from within my library B for security reasons. What if it unwinds the stack, connects to A's HQ, and dumps my confidential memory all over there? >>> >>> Yikes! ï¿½Don't link with libraries you don't trust :-) >>> >>> I don't know how to handle this case, but does the current standard prevent a library from snooping memory from other libraries? ï¿½A library could set an attribute with a copy callback function on comm_world. ï¿½That would be called from within another library's stack if that library tries to dup comm_world. >>> >>>> Moreover, by the time it's called, both A and commA may be the thing of the times long gone together with the context in which handlerA was supposed to be executed. What will it try to handle then and under what assumptions? I don't know. You? >>> >>> The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. >>> >>> -d >>> >>> >>>> -d >>>> >>>> >>>> On Oct 21, 2011, at 2:42 PM, Supalov, Alexander wrote: >>>> >>>>> Not really. How do you want the user make sense of that? E.g., I call A on commA, fail on commA asynchronously while calling a totally unrelated B on commB that has no failures in it, and am kicked out of B into someone else's error handler saying some "A" on "comma" failed? And what now? I may even have A and commA out of scope by then, possibly forever. >>>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>> Sent: Friday, October 21, 2011 9:35 PM >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>> >>>>> With (regular) error handlers, they'll be called from within the function that raises the error. ï¿½With failure notification, because they're being called as a result of an external event (process failure), you could be called from within any function, even one not related to the comm/file/win that you registered the process failure notification handler on. >>>>> >>>>> Does that make sense? >>>>> >>>>> -d >>>>> >>>>> On Oct 21, 2011, at 1:51 PM, Sur, Sayantan wrote: >>>>> >>>>>> 17.5.1:11-12 - "The error handler function will be called by the MPI implementation from within the context of some MPI function that was called by the user." >>>>>> >>>>>> Maybe we should that error handlers are called from MPI functions that are associated with that comm/file/win? >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>>>>>> Sent: Friday, October 21, 2011 10:28 AM >>>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>>> >>>>>>> I just wanted to note that we want to distribute a copy of this >>>>>>> chapter to the MPI Forum before the meeting. As such we are planning >>>>>>> on sending out a copy at COB today (so Friday ~5:00 pm EDT) so that >>>>>>> people have an opportunity to look at the document before the Monday >>>>>>> plenary. So please send any edits or comments before COB today, so we >>>>>>> can work them into the draft. >>>>>>> >>>>>>> We will post the draft to the ticket, so that people know where to >>>>>>> look for the current draft. >>>>>>> >>>>>>> Thanks, >>>>>>> Josh >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 20, 2011 at 7:06 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> >>>>>>> wrote: >>>>>>>> >>>>>>>> Here's the latest version of the FT chapter is on the wiki (it's in a >>>>>>> new location on the main FT page under "ticket #276". ï¿½Please have a >>>>>>> look and comment. >>>>>>>> >>>>>>>> Here's a direct link to the PDF: >>>>>>>> ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/raw- >>>>>>> attachment/wiki/FaultToleranceWikiPage/ft.pdf >>>>>>>> >>>>>>>> Here's a summary of the changes Josh and I made: >>>>>>>> >>>>>>>> * Minor wording touchups >>>>>>>> * Added new semantic for MPI_ANY_SOURCE with the >>>>>>> MPI_ERR_ANY_SOURCE_DISABLED error code >>>>>>>> * Coverted wording for all comm, win, fh creation operations to not >>>>>>> require collectively active communicators (eliminate requirement for >>>>>>> synchronization) >>>>>>>> * Added missing reader_lock to ANY_SOURCE example >>>>>>>> * Added case for MPI_WIN_TEST >>>>>>>> >>>>>>>> and >>>>>>>> >>>>>>>> One-sided section >>>>>>>> ï¿½clarified that window creation need not be blocking >>>>>>>> ï¿½clarified that RMA ops might not complete correctly even if >>>>>>>> ï¿½ ï¿½synchronization ops complete without error due to process >>>>>>>> ï¿½ ï¿½failures >>>>>>>> Process failure notification >>>>>>>> ï¿½Added section describing new functions to add callbacks to comms, >>>>>>>> ï¿½ ï¿½wins and files that are called when proc failure is detected >>>>>>>> Other wordsmithing/cleanup changes >>>>>>>> >>>>>>>> -d >>>>>>>> _______________________________________________ >>>>>>>> mpi3-ft mailing list >>>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Joshua Hursey >>>>>>> Postdoctoral Research Associate >>>>>>> Oak Ridge National Laboratory >>>>>>> http://users.nccs.gov/~jjhursey >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> -------------------------------------------------------------------------------------- >>>>> Intel GmbH >>>>> Dornacher Strasse 1 >>>>> 85622 Feldkirchen/Muenchen, Deutschland >>>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>>> Registergericht: Muenchen HRB 47456 >>>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> -------------------------------------------------------------------------------------- >>>> Intel GmbH >>>> Dornacher Strasse 1 >>>> 85622 Feldkirchen/Muenchen, Deutschland >>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>> Registergericht: Muenchen HRB 47456 >>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> -------------------------------------------------------------------------------------- >>> Intel GmbH >>> Dornacher Strasse 1 >>> 85622 Feldkirchen/Muenchen, Deutschland >>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>> Registergericht: Muenchen HRB 47456 >>> Ust.-IdNr./VAT Registration No.: DE129385895 >>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] one sided
by Sur, Sayantan 21 Oct '11

21 Oct '11

Thanks for the clarification, Josh. It makes sense to me. > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey > Sent: Friday, October 21, 2011 2:28 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] one sided > > Since MPI is creating a new communication object, we require that all > processes have access to the object if it was created anywhere. > > So consider if the 'win' object was created at some processes and not > others. The application with the valid 'win' calls MPI_Win_fence(). > Other processes in the group associated with the window will not be to > call MPI_Win_fence since they do not have a valid object. The program > is erroneous since not all processes are calling the collective. So > the semantics become muddled when we talk about collective operations > (even like MPI_Win_free) when not all processes are guaranteed to have > a valid communication object to use. > > So we just need the requirement that the object is either created > everywhere or nowhere. Since we can only make statements about the > behavior of MPI after the MPI_ERR_PROC_FAIL_STOP error code, we > restrict the language to just that error code. Though it could be > argued that this is a more general requirement, but that is slightly > out of scope for this proposal. > > Does that help clarify? > > -- Josh > > On Fri, Oct 21, 2011 at 4:54 PM, Sur, Sayantan <sayantan.sur(a)intel.com> > wrote: > > Hi All, > > > > The new chapter says this about the window creation: > > > > "If the MPI_WIN_CREATE operation fails at any live process due to a > process failure, then the operation must fail at every live process > with an error in the class MPI_ERR_PROC_FAIL_STOP." > > > > I'm wondering what would happen if MPI_WIN_CREATE did not have this > qualification at all. i.e. it would succeed at some processes and fail > at some processes. After all, any following GET or PUT calls can always > raise the error class MPI_ERR_PROC_FAIL_STOP. Also, the communicator > passed to MPI_WIN_CREATE is allowed to have dead processes in it ... > then why qualify win create with this requirement? > > > > Thanks, > > Sayantan. > > > >> -----Original Message----- > >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > >> bounces(a)lists.mpi-forum.org] On Behalf Of Pavan Balaji > >> Sent: Wednesday, October 19, 2011 9:05 AM > >> To: mpi3-ft(a)lists.mpi-forum.org > >> Subject: Re: [Mpi3-ft] one sided > >> > >> > >> FYI, you cannot "require" some behavior from MPI through an info > >> argument. It is perfectly legitimate for the MPI implementation to > >> completely ignore any info arguments passed. They are just user > hints. > >> > >> ï¿½ -- Pavan > >> > >> On 10/19/2011 07:59 AM, Josh Hursey wrote: > >> > Let's be sure to talk about this on today's call. I have some > other > >> > one-sided notes that I would like to go over as well. > >> > > >> > It would be fairly easy to support both modes since the > >> MPI_Win_create > >> > operation takes an info argument. We could define a key (similar > to > >> > what they have done for other operations) that either loosens or > >> > tightens the semantics depending on what the default behavior > should > >> > be. > >> > > >> > I think it is ok to have a non-synchronizing option, just as long > as > >> > we have clear semantics for when the window is not created at all > >> > processes due to some process failure - or if the window is always > >> > created regardless of emerging failure then we might avoid this > >> issue, > >> > but that might require some additional clarification. > >> > > >> > Thanks, > >> > Josh > >> > > >> > On Wed, Oct 19, 2011 at 4:11 AM, Supalov, Alexander > >> > <alexander.supalov(a)intel.com> ï¿½wrote: > >> >> Thanks. Why not having two calls or modes of operation to cover > >> both? > >> >> > >> >> -----Original Message----- > >> >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > >> bounces(a)lists.mpi-forum.org] On Behalf Of Darius Buntinas > >> >> Sent: Tuesday, October 18, 2011 9:57 PM > >> >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >> Group > >> >> Subject: [Mpi3-ft] one sided > >> >> > >> >> > >> >> I got some feedback from Jim and Pavan on the one-sided section. > >> One thing Jim pointed out was that we don't want to make window > >> creation synchronizing, and the fail-or-succeed everywhere > requirement > >> would do that. > >> >> > >> >> If we say that window creation should not fail due to failed > >> processes, that would accomplish the same thing: ï¿½If a window is > >> created by a correct program, then it will succeed at all live > >> processes. ï¿½Note that if an incorrect program specifies invalid > >> parameters then the window creation may fail at some processes and > >> succeed at others, but this is what we already have today. > >> >> > >> >> However, it's possible that some implementations cannot satisfy > this > >> requirement because, e.g., they do collectives as part of the > >> operation. ï¿½So maybe we should have two options: > >> >> > >> >> ï¿½ Either: > >> >> ï¿½ ï¿½ window creation won't fail because if failed processes > >> >> ï¿½ or > >> >> ï¿½ ï¿½ window creation will either succeed or fail everywhere and if > >> window creation fails at > >> >> ï¿½ ï¿½ any process it fails at every process > >> >> > >> >> -d > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> mpi3-ft mailing list > >> >> mpi3-ft(a)lists.mpi-forum.org > >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >> >> ----------------------------------------------------------------- > --- > >> ------------------ > >> >> Intel GmbH > >> >> Dornacher Strasse 1 > >> >> 85622 Feldkirchen/Muenchen, Deutschland > >> >> Sitz der Gesellschaft: Feldkirchen bei Muenchen > >> >> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes > Schwaderer > >> >> Registergericht: Muenchen HRB 47456 > >> >> Ust.-IdNr./VAT Registration No.: DE129385895 > >> >> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 > >> >> > >> >> > >> >> _______________________________________________ > >> >> mpi3-ft mailing list > >> >> mpi3-ft(a)lists.mpi-forum.org > >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >> >> > >> >> > >> > > >> > > >> > > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > >> _______________________________________________ > >> mpi3-ft mailing list > >> mpi3-ft(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Latest version of chapter
by Darius Buntinas 21 Oct '11

21 Oct '11

On Oct 21, 2011, at 4:18 PM, Josh Hursey wrote: > The original problem was that the application wanted uniform > notification of a process failure (restricted to the set of processes > in the group associated with the communication objection). The current > error handlers are only fired when interacting with the failed process > directly (P2P) or indirectly (ANY_SOURCE, collectives). > > The requesters (who I believe are on the list and may want to pipe up) > were ok with having the callback triggered at an MPI boundary - so not > really asynchronous just not associated with the call. > > Maybe it is enough to restrict the notification to operations on the > communicator. So the FailHandler registered on commA is only fired > when commA is being used. The application would have to register the > FailHandler on all communicators that it is using and wants > notification from. But that would preserve some separation between the > library and application. I think that makes sense. > > A few things that probably should be clarified with the new FailHandler: > - Is it inherited by new communicators like other error handlers? I'd say no. Because all it would do is call the same handler once for every communicator for the failure of the same process. > - Without the communicator scope restriction mentioned above, if a > process fails, does it fire all of the FailHanders registered on > communication objects containing that process? (I think yes) If so, we > should probably state that we do not guarantee any ordering of these > calls. If we restrict it to only be called from an operation that uses that comm/win/file, then it would only fire one handler. If multiple failures happen since the last time you made a call with a particular comm/win/file, then it should only be called once (not once per failure), because the user can find out about all failed processes at that time. > - In the function signatures the errhandler is 'int'/'integer' and > should probably be MPI_Errhandler or similar handle. I copied the prototypes from the error handler section. > - On the topic of what functions you can use inside, we can probably > use the language from the error handlers. I think it allows the user > to do pretty much anything they want, though I'd have to double check. > It might be that the standard is silent on this point, so no specific > restrictions are defined. I didn't see any restrictions, but then the standard says that all bets are off when you get an error, so calling anything at that point is undefined. -d > > > -- Josh > > > On Fri, Oct 21, 2011 at 4:54 PM, Supalov, Alexander > <alexander.supalov(a)intel.com> wrote: >> Imagine I use some data protection scheme inside B. I won't be affected by "wrong" libraries that I call before or after my protection is on. I may be affected by an asynchronous call out of "another world" that is possible if handlerA is called from within my library B. I.e., in the sequence >> >> A-B-A >> >> this extension allows B to be "hacked" by A by just killing one process at the right time. Moreover, I can clean up the callbacks by using MPI_Comm_create instead of MPI_Comm_dup. I cannot prevent an asynchronous handler from being called. >> >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >> Sent: Friday, October 21, 2011 10:44 PM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: Re: [Mpi3-ft] Latest version of chapter >> >> >> On Oct 21, 2011, at 3:19 PM, Supalov, Alexander wrote: >> >>> Thanks. See below (prefix "AS>"). >>> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>> Sent: Friday, October 21, 2011 9:57 PM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> Subject: Re: [Mpi3-ft] Latest version of chapter >>> >>> >>> Let's say you have commA used by library A and commB used by library B. >>> Library A has registered the proc failure handler called handlerA on commA. >>> >>> Now, let's say a process that's in commA but not commB failed, and the thread is executing in library B and calls, e.g., MPI_Send(..., commB). >>> >>> The MPI implementation performs the MPI_Send operation normally, then calls handlerA(commA, MPI_ERR_PROC_FAIL_STOP), and returns from MPI_Send normally. >>> >>> While in handlerA, the subject communicator (commA) is passed as a parameter, so it won't be out of scope. >>> >>> Is it a problem that library A's handler is called from "within" library B? >>> >>> AS> Sure. This handler may have been written by someone else who does not know me or my B or anything else. I may not even want it to be called from within my library B for security reasons. What if it unwinds the stack, connects to A's HQ, and dumps my confidential memory all over there? >> >> Yikes! Don't link with libraries you don't trust :-) >> >> I don't know how to handle this case, but does the current standard prevent a library from snooping memory from other libraries? A library could set an attribute with a copy callback function on comm_world. That would be called from within another library's stack if that library tries to dup comm_world. >> >>> Moreover, by the time it's called, both A and commA may be the thing of the times long gone together with the context in which handlerA was supposed to be executed. What will it try to handle then and under what assumptions? I don't know. You? >> >> The handlers are freed when the comm/win/file they're attached to is freed, so you'll never get a handler called with a comm/win/file that's invalid. >> >> -d >> >> >>> -d >>> >>> >>> On Oct 21, 2011, at 2:42 PM, Supalov, Alexander wrote: >>> >>>> Not really. How do you want the user make sense of that? E.g., I call A on commA, fail on commA asynchronously while calling a totally unrelated B on commB that has no failures in it, and am kicked out of B into someone else's error handler saying some "A" on "comma" failed? And what now? I may even have A and commA out of scope by then, possibly forever. >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>> Sent: Friday, October 21, 2011 9:35 PM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>> >>>> With (regular) error handlers, they'll be called from within the function that raises the error. With failure notification, because they're being called as a result of an external event (process failure), you could be called from within any function, even one not related to the comm/file/win that you registered the process failure notification handler on. >>>> >>>> Does that make sense? >>>> >>>> -d >>>> >>>> On Oct 21, 2011, at 1:51 PM, Sur, Sayantan wrote: >>>> >>>>> 17.5.1:11-12 - "The error handler function will be called by the MPI implementation from within the context of some MPI function that was called by the user." >>>>> >>>>> Maybe we should that error handlers are called from MPI functions that are associated with that comm/file/win? >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >>>>>> bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey >>>>>> Sent: Friday, October 21, 2011 10:28 AM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>>> Subject: Re: [Mpi3-ft] Latest version of chapter >>>>>> >>>>>> I just wanted to note that we want to distribute a copy of this >>>>>> chapter to the MPI Forum before the meeting. As such we are planning >>>>>> on sending out a copy at COB today (so Friday ~5:00 pm EDT) so that >>>>>> people have an opportunity to look at the document before the Monday >>>>>> plenary. So please send any edits or comments before COB today, so we >>>>>> can work them into the draft. >>>>>> >>>>>> We will post the draft to the ticket, so that people know where to >>>>>> look for the current draft. >>>>>> >>>>>> Thanks, >>>>>> Josh >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 20, 2011 at 7:06 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> >>>>>> wrote: >>>>>>> >>>>>>> Here's the latest version of the FT chapter is on the wiki (it's in a >>>>>> new location on the main FT page under "ticket #276". Please have a >>>>>> look and comment. >>>>>>> >>>>>>> Here's a direct link to the PDF: >>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/raw- >>>>>> attachment/wiki/FaultToleranceWikiPage/ft.pdf >>>>>>> >>>>>>> Here's a summary of the changes Josh and I made: >>>>>>> >>>>>>> * Minor wording touchups >>>>>>> * Added new semantic for MPI_ANY_SOURCE with the >>>>>> MPI_ERR_ANY_SOURCE_DISABLED error code >>>>>>> * Coverted wording for all comm, win, fh creation operations to not >>>>>> require collectively active communicators (eliminate requirement for >>>>>> synchronization) >>>>>>> * Added missing reader_lock to ANY_SOURCE example >>>>>>> * Added case for MPI_WIN_TEST >>>>>>> >>>>>>> and >>>>>>> >>>>>>> One-sided section >>>>>>> clarified that window creation need not be blocking >>>>>>> clarified that RMA ops might not complete correctly even if >>>>>>> synchronization ops complete without error due to process >>>>>>> failures >>>>>>> Process failure notification >>>>>>> Added section describing new functions to add callbacks to comms, >>>>>>> wins and files that are called when proc failure is detected >>>>>>> Other wordsmithing/cleanup changes >>>>>>> >>>>>>> -d >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Joshua Hursey >>>>>> Postdoctoral Research Associate >>>>>> Oak Ridge National Laboratory >>>>>> http://users.nccs.gov/~jjhursey >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> -------------------------------------------------------------------------------------- >>>> Intel GmbH >>>> Dornacher Strasse 1 >>>> 85622 Feldkirchen/Muenchen, Deutschland >>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>>> Registergericht: Muenchen HRB 47456 >>>> Ust.-IdNr./VAT Registration No.: DE129385895 >>>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> -------------------------------------------------------------------------------------- >>> Intel GmbH >>> Dornacher Strasse 1 >>> 85622 Feldkirchen/Muenchen, Deutschland >>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>> Registergericht: Muenchen HRB 47456 >>> Ust.-IdNr./VAT Registration No.: DE129385895 >>> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> -------------------------------------------------------------------------------------- >> Intel GmbH >> Dornacher Strasse 1 >> 85622 Feldkirchen/Muenchen, Deutschland >> Sitz der Gesellschaft: Feldkirchen bei Muenchen >> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >> Registergericht: Muenchen HRB 47456 >> Ust.-IdNr./VAT Registration No.: DE129385895 >> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] one sided
by Josh Hursey 21 Oct '11

21 Oct '11

Since MPI is creating a new communication object, we require that all processes have access to the object if it was created anywhere. So consider if the 'win' object was created at some processes and not others. The application with the valid 'win' calls MPI_Win_fence(). Other processes in the group associated with the window will not be to call MPI_Win_fence since they do not have a valid object. The program is erroneous since not all processes are calling the collective. So the semantics become muddled when we talk about collective operations (even like MPI_Win_free) when not all processes are guaranteed to have a valid communication object to use. So we just need the requirement that the object is either created everywhere or nowhere. Since we can only make statements about the behavior of MPI after the MPI_ERR_PROC_FAIL_STOP error code, we restrict the language to just that error code. Though it could be argued that this is a more general requirement, but that is slightly out of scope for this proposal. Does that help clarify? -- Josh On Fri, Oct 21, 2011 at 4:54 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: > Hi All, > > The new chapter says this about the window creation: > > "If the MPI_WIN_CREATE operation fails at any live process due to a process failure, then the operation must fail at every live process with an error in the class MPI_ERR_PROC_FAIL_STOP." > > I'm wondering what would happen if MPI_WIN_CREATE did not have this qualification at all. i.e. it would succeed at some processes and fail at some processes. After all, any following GET or PUT calls can always raise the error class MPI_ERR_PROC_FAIL_STOP. Also, the communicator passed to MPI_WIN_CREATE is allowed to have dead processes in it ... then why qualify win create with this requirement? > > Thanks, > Sayantan. > >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Pavan Balaji >> Sent: Wednesday, October 19, 2011 9:05 AM >> To: mpi3-ft(a)lists.mpi-forum.org >> Subject: Re: [Mpi3-ft] one sided >> >> >> FYI, you cannot "require" some behavior from MPI through an info >> argument. It is perfectly legitimate for the MPI implementation to >> completely ignore any info arguments passed. They are just user hints. >> >> ï¿½ -- Pavan >> >> On 10/19/2011 07:59 AM, Josh Hursey wrote: >> > Let's be sure to talk about this on today's call. I have some other >> > one-sided notes that I would like to go over as well. >> > >> > It would be fairly easy to support both modes since the >> MPI_Win_create >> > operation takes an info argument. We could define a key (similar to >> > what they have done for other operations) that either loosens or >> > tightens the semantics depending on what the default behavior should >> > be. >> > >> > I think it is ok to have a non-synchronizing option, just as long as >> > we have clear semantics for when the window is not created at all >> > processes due to some process failure - or if the window is always >> > created regardless of emerging failure then we might avoid this >> issue, >> > but that might require some additional clarification. >> > >> > Thanks, >> > Josh >> > >> > On Wed, Oct 19, 2011 at 4:11 AM, Supalov, Alexander >> > <alexander.supalov(a)intel.com> ï¿½wrote: >> >> Thanks. Why not having two calls or modes of operation to cover >> both? >> >> >> >> -----Original Message----- >> >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Darius Buntinas >> >> Sent: Tuesday, October 18, 2011 9:57 PM >> >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >> Group >> >> Subject: [Mpi3-ft] one sided >> >> >> >> >> >> I got some feedback from Jim and Pavan on the one-sided section. >> One thing Jim pointed out was that we don't want to make window >> creation synchronizing, and the fail-or-succeed everywhere requirement >> would do that. >> >> >> >> If we say that window creation should not fail due to failed >> processes, that would accomplish the same thing: ï¿½If a window is >> created by a correct program, then it will succeed at all live >> processes. ï¿½Note that if an incorrect program specifies invalid >> parameters then the window creation may fail at some processes and >> succeed at others, but this is what we already have today. >> >> >> >> However, it's possible that some implementations cannot satisfy this >> requirement because, e.g., they do collectives as part of the >> operation. ï¿½So maybe we should have two options: >> >> >> >> ï¿½ Either: >> >> ï¿½ ï¿½ window creation won't fail because if failed processes >> >> ï¿½ or >> >> ï¿½ ï¿½ window creation will either succeed or fail everywhere and if >> window creation fails at >> >> ï¿½ ï¿½ any process it fails at every process >> >> >> >> -d >> >> >> >> >> >> >> >> _______________________________________________ >> >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> -------------------------------------------------------------------- >> ------------------ >> >> Intel GmbH >> >> Dornacher Strasse 1 >> >> 85622 Feldkirchen/Muenchen, Deutschland >> >> Sitz der Gesellschaft: Feldkirchen bei Muenchen >> >> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >> >> Registergericht: Muenchen HRB 47456 >> >> Ust.-IdNr./VAT Registration No.: DE129385895 >> >> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 >> >> >> >> >> >> _______________________________________________ >> >> mpi3-ft mailing list >> >> mpi3-ft(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> >> >> > >> > >> > >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0