January 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Fault Tolerance & I/O Discussion
by Josh Hursey 31 Jan '12

31 Jan '12

Sorry about that. I forgot to set the timezone when setting up the poll, and it looks like I cannot change it now. So I created a new poll. Please use the link below, it should now allow you to adjust the time zone local via a drop down menu at the top of the poll. http://www.doodle.com/yifhpi5emyyzrspa Thanks for letting me know. -- Josh On Tue, Jan 31, 2012 at 3:44 PM, Dries Kimpe <dkimpe(a)mcs.anl.gov> wrote: > What's the time zone for the times listed in the doodle poll? > Or does doodle automatically adjust? > > Dries > > * Josh Hursey <jjhursey(a)open-mpi.org> [2012-01-31 12:52:58]: > > > (Cross posted to both the I/O and FT MPI-3 listservs) > > > During the FT plenary session at the Jan. MPI Forum meeting there were > some > > concerns about fault tolerance semantics in the I/O chapter. We did not > > have much time to fully discuss the additional semantics during the > > meeting. To make sure that we push towards a complete set of semantics > > useful for the I/O community in the next draft I would like to have a > > teleconf to discuss the I/O chapter of the FT proposal. Preferability in > > the next week and a half (so we have time to fine tune things before the > > next forum meeting). > > > Below is a link to a doodle poll to find a good time for a teleconf. If > you > > are interested in participating in this discussion, please fill this poll > > out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. > > http://www.doodle.com/s3hz9daeh8pn483m > > > Thanks, > > Josh > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Fault Tolerance & I/O Discussion
by Dries Kimpe 31 Jan '12

31 Jan '12

What's the time zone for the times listed in the doodle poll? Or does doodle automatically adjust? Dries * Josh Hursey <jjhursey(a)open-mpi.org> [2012-01-31 12:52:58]: > (Cross posted to both the I/O and FT MPI-3 listservs) > During the FT plenary session at the Jan. MPI Forum meeting there were some > concerns about fault tolerance semantics in the I/O chapter. We did not > have much time to fully discuss the additional semantics during the > meeting. To make sure that we push towards a complete set of semantics > useful for the I/O community in the next draft I would like to have a > teleconf to discuss the I/O chapter of the FT proposal. Preferability in > the next week and a half (so we have time to fine tune things before the > next forum meeting). > Below is a link to a doodle poll to find a good time for a teleconf. If you > are interested in participating in this discussion, please fill this poll > out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. > http://www.doodle.com/s3hz9daeh8pn483m > Thanks, > Josh

1 0

Re: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 31 Jan '12

31 Jan '12

We have a (over) full call for this week. The next call will be Feb. 8, though we will discuss this date/time further during the teleconf tomorrow. -- Josh On Tue, Jan 31, 2012 at 1:12 PM, Anthony Skjellum <tony(a)cis.uab.edu> wrote: > Josh, we will be ready to present our work at the next call; will that be > Feb 8 or later? > Tony > > ----- Original Message ----- > From: "Josh Hursey" <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" < > mpi3-ft(a)lists.mpi-forum.org> > Sent: Tuesday, January 31, 2012 11:54:39 AM > Subject: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST > > > > Reminder that we have a teleconf tomorrow, Wednesday, at Noon EST. > > > Date: February 1, 2012 > Time: Noon EST/New York > Dial-in information: 877-801-8130 > Code: 1044056 > > > > > Agenda: > - Misc. > - RMA Teleconf (Feb 2 @ Noon Eastern) > - I/O Teleconf (TBD) > - Doodle Poll: http://www.doodle.com/s3hz9daeh8pn483m > - Timeline > - March 6 - MPI Forum Meeting (first reading) > - Feb. 20 - Revised proposal due (-2 weeks) > - Feb. 13 - Draft due for working group review (-3 weeks) > - Feb. 6 - Firm decisions on pending issues (-4 weeks) > - Possibly shifting time/date of meeting next week. > - Piggybacking (Martin S.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - Run-Through Stabilization Proposal > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… > http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php > - Discuss FailHandlers > - Discuss communicator creation matching > - Decide on Collectives over communicators with holes > - Decide on MPI_ANY_SOURCE semantic > - Other items? > > > > > Additional Agenda Items [as time permits]: > - Process Recovery (Josh H.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 > - MPI Forum notes discussion (?) > http://lists.mpi-forum.org/mpi3-ft/2012/01/0983.php > http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php > - Error to Fault changes proposal (Darius B.) > http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php > - Network Channel Failure > http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php > - FT Query Interface > http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php > > > > > Next Meetings: > * Feb. 1, 2012 (Teleconf) > * Feb. 2, 2012 (Teleconf - RMA Working Group) > * Feb. 8, 2012 (Teleconf - As needed) > * Feb. 15, 2012 (Teleconf) > * Feb. 22, 2012 (Teleconf - As needed) > * Feb. 29, 2012 (Teleconf) > * March 5, 2012 (MPI Forum Meeting) > * March 14, 2012 (Teleconf) > > > Respond to the list if you have any additional topics that you would like > to discuss. > > > -- Josh > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > Anthony Skjellum, PhD > Professor and Chair > Dept. of Computer and Information Sciences > Director, UAB Center for Information Assurance and Joint Forensics > Research ("The Center") > University of Alabama at Birmingham > +1-(205)934-8657; FAX: +1- (205)934-5473 > > ___________________________________________ > CONFIDENTIALITY: This e-mail and any attachments are confidential and > may be privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person, > use it for any purpose or store or copy the information in any medium. > > Please consider the environment before printing this e-mail > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Anthony Skjellum 31 Jan '12

31 Jan '12

Josh, we will be ready to present our work at the next call; will that be Feb 8 or later? Tony ----- Original Message ----- From: "Josh Hursey" <jjhursey(a)open-mpi.org> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Sent: Tuesday, January 31, 2012 11:54:39 AM Subject: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST Reminder that we have a teleconf tomorrow, Wednesday, at Noon EST. Date: February 1, 2012 Time: Noon EST/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Misc. - RMA Teleconf (Feb 2 @ Noon Eastern) - I/O Teleconf (TBD) - Doodle Poll: http://www.doodle.com/s3hz9daeh8pn483m - Timeline - March 6 - MPI Forum Meeting (first reading) - Feb. 20 - Revised proposal due (-2 weeks) - Feb. 13 - Draft due for working group review (-3 weeks) - Feb. 6 - Firm decisions on pending issues (-4 weeks) - Possibly shifting time/date of meeting next week. - Piggybacking (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php - Discuss FailHandlers - Discuss communicator creation matching - Decide on Collectives over communicators with holes - Decide on MPI_ANY_SOURCE semantic - Other items? Additional Agenda Items [as time permits]: - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2012/01/0983.php http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Feb. 1, 2012 (Teleconf) * Feb. 2, 2012 (Teleconf - RMA Working Group) * Feb. 8, 2012 (Teleconf - As needed) * Feb. 15, 2012 (Teleconf) * Feb. 22, 2012 (Teleconf - As needed) * Feb. 29, 2012 (Teleconf) * March 5, 2012 (MPI Forum Meeting) * March 14, 2012 (Teleconf) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 31 Jan '12

31 Jan '12

Reminder that we have a teleconf tomorrow, Wednesday, at Noon EST. Date: February 1, 2012 Time: Noon EST/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Misc. - RMA Teleconf (Feb 2 @ Noon Eastern) - I/O Teleconf (TBD) - Doodle Poll: http://www.doodle.com/s3hz9daeh8pn483m - Timeline - March 6 - MPI Forum Meeting (first reading) - Feb. 20 - Revised proposal due (-2 weeks) - Feb. 13 - Draft due for working group review (-3 weeks) - Feb. 6 - Firm decisions on pending issues (-4 weeks) - Possibly shifting time/date of meeting next week. - Piggybacking (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php - Discuss FailHandlers - Discuss communicator creation matching - Decide on Collectives over communicators with holes - Decide on MPI_ANY_SOURCE semantic - Other items? Additional Agenda Items [as time permits]: - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2012/01/0983.php http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Feb. 1, 2012 (Teleconf) * Feb. 2, 2012 (Teleconf - RMA Working Group) * Feb. 8, 2012 (Teleconf - As needed) * Feb. 15, 2012 (Teleconf) * Feb. 22, 2012 (Teleconf - As needed) * Feb. 29, 2012 (Teleconf) * March 5, 2012 (MPI Forum Meeting) * March 14, 2012 (Teleconf) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Fault Tolerance & I/O Discussion
by Josh Hursey 31 Jan '12

31 Jan '12

(Cross posted to both the I/O and FT MPI-3 listservs) During the FT plenary session at the Jan. MPI Forum meeting there were some concerns about fault tolerance semantics in the I/O chapter. We did not have much time to fully discuss the additional semantics during the meeting. To make sure that we push towards a complete set of semantics useful for the I/O community in the next draft I would like to have a teleconf to discuss the I/O chapter of the FT proposal. Preferability in the next week and a half (so we have time to fine tune things before the next forum meeting). Below is a link to a doodle poll to find a good time for a teleconf. If you are interested in participating in this discussion, please fill this poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. http://www.doodle.com/s3hz9daeh8pn483m Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Matching MPI communication object creation across process failure
by Josh Hursey 31 Jan '12

31 Jan '12

Actually the following thread might be more useful for this discussion: http://lists.mpi-forum.org/mpi3-ft/2011/12/0940.php The example did not come out well in the archives, so below is the diagram again (hopefully that will work): So the process stack looks like: P0 P1 --------------- ---------------- Dup(comm[X-1]) Dup(comm[X-1]) MPI_Allreduce() MPI_Allreduce() Dup(comm[X]) -> Error -> Error So should P1 be required to call Dup(comm[X])? -- Josh On Wed, Jan 25, 2012 at 5:08 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > The current proposal states that MPI object creation functions (e.g., > MPI_Comm_create, MPI_Win_create, MPI_File_open): > ------------------- > All participating communicator(s) must be collectively active before > calling any communicator creation operation. Otherwise, the communicator > creation operation will uniformly raise an error code of the class > MPI_ERR_PROC_FAIL_STOP. > > If a process failure prevents the uniform creation of the communicator > then the communicator construction operation must ensure that the > communicator is not created, and all alive participating processes will > raise an error code of the class MPI_ERR_PROC_FAIL_STOP. Communicator > construction operations will match across the notification of a process > failure. As such, all alive processes must call the communicator > construction operations the same number of times regardless of whether the > emergent process failure makes the call irrelevant to the application. > ------------------- > > So there are three points here: > (1) That the communicator must be 'collectively active' before calling > the operation, > (2) Uniform creation of the communication object, and > (3) Creation operations match across process failure. > > Point (2) seems to be necessary so that all processes only ever see a > communication object that is consistent across all processes. This implies > a fault tolerant agreement protocol (group membership). > > There was a question about why point (3) is necessary. We (Darius, UTK, > and I) discussed this on 12/12/2011, and I posted my notes on this to the > list: > http://lists.mpi-forum.org/mpi3-ft/2011/12/0938.php > Looking back at my written notes, they don't have much more than that to > add to the discussion. > > So the problem with (3) seemed to arrise out of the failure handlers, > though I am not convinced that they are strictly to blame in this > circumstance. It seems that the agreement protocol might be factoring into > the discussion as well since it is strongly synchronizing, if not all > processes call the operation how does it know when to bail out. The peer > processes are (a) calling that operation, (b) going to call it but have not > yet, or (c) will never call it because they decided independently not to > base on a locally reported process failure. > > It seems that the core problem has to do with when to break out of the > collective creation operation, and when to restore matching. > > So should re reconsider the restriction on (3)? More to the point, is it > safe to not require (3)? > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by George Bosilca 28 Jan '12

28 Jan '12

-- This comment is coming too late for the discussion on the FT call, but I'll leave it here for further clarification of the issue with SHRINK. -- To be precise it depends on the communication mode, reset or continue. However, such concerns are technicalities. The real problem with SHRINK is that it cannot work properly if you do not break all existing communicators, as the local view of several software layers can be inconsistent. In the context of FT-MPI, where all communicators were marked as invalid (on which any subsequent MPI call was bound to fail) on any [asynchronous] detection of lost processes, this approach make sense. In the context of the current proposal and discussion in the forum, where the same communicators can be fixed this approach is in my view very difficult to defend. george. On Jan 24, 2012, at 15:09 , Josh Hursey wrote: > It was also mentioned to me that the SHRINK mode also forces all messages to be dropped when the communicator is repaired. Shrinking will likely shift the ranks of processes in the communicator (to fill in the gaps) so matching will need to be reset, and messages would need to be dropped/drained. So it would no longer be possible for an application re-enable collectives and still have messages in-flight across the repair. At least not without further clarification on this point. > > -- Josh > > On Tue, Jan 24, 2012 at 2:41 PM, Graham, Richard L. <rlgraham(a)ornl.gov> wrote: > I will re-iterate what I said before. While this may one mode that apps may want to use, it is not the only mode. In particular, this forces all ranks in a communicator to know about the change, even if they have implemented a "local" algorithm that does not need to know about all failures. > > Rich > > On Jan 24, 2012, at 2:09 PM, Sur, Sayantan wrote: > > Hi Josh, > > Thanks for the crisp characterization of the proposal I was making. It is correct. I was naturally thinking of the SHRINK mode, since it involves least number of changes to MPI standard itself. Folks at the forum also had similar thoughts (e.g. why does MPI_Comm_size() still return count including failed procs). > > Cf. http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT-MPI.pdf > > “4.2 FTMPI_COMM_MODE_SHRINK > > In this communicator mode, the ranks of MPI processes before and after recovery might change, as well as the size of MPI COMM WORLD does change. The appealing part of this communicator mode however is, that all functions specified in MPI-1 and MPI-2 are still valid without any further modification, since groups and communicators do not have wholes (sic) and blank processes.” > > We can discuss further tomorrow as to whether we could go with SHRINK mode (simplifying the proposal). From what I read in the paper, they report being able to convert fault-tolerant master/worker to use both modes. > > Thanks! > > === > Sayantan Sur, Ph.D. > Intel Corp. > > From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Josh Hursey > Sent: Tuesday, January 24, 2012 10:39 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal > > First let me say that I greatly appreciate the effort of Sayantan and others to push us towards considering alternative techniques, and stimulating discussion about design decisions. This is exactly the type of discussion that needs to occur, and the working group is the most appropriate place to have it. > > > One of the core suggestions of Sayantan's proposal is the switch from (using FT-MPI's language) a model like BLANK to a model like SHRINK. I think many of the other semantics are derived from this core shift, so we should probably focus the discussion on this point earlier in our conversation. > > > The current RTS proposal allows for a communicator to contain failed processes and continue to be used for all operations, including collectives, after acknowledging them. This matches closely to FT-MPI's BLANK mode. The user can use MPI_Comm_split() to get the equivalent of SHRINK if they need it. > > The suggested modification allows for only/primarily a SHRINK-like mode in order to have full functionality in the communicator. As discussed on the previous call, one can get the BLANK mode by adding a library on top of MPI that virtualizes the communicators to create shadow communicators. The argument for the SHRINK mode is that it is -easier- to pass/explain. > > The reason we chose BLANK was derived from the literature reviewed, code examples available, and feedback from application groups. >From which there seemed to be a strong demand for the BLANK mode. In fact, I had a difficult time finding good use cases for the SHIRNK mode (I'm still looking though). Additionally, a BLANK mode seems also to make it easier to reason about process recovery. To reason about process recovery (something like FT-MPI's REBUILD mode) one needs to be able to reason about the missing processes without changing the identities of the existing processes, which can be difficult in a SHRINK mode. So from this review it seemed that there was an application demand for a BLANK-like mode for the RTS proposal. > > In light of this background, it is concerning to me to advise these application users that MPI will not provide the functionality they require, but they have to depend upon a non-standard, third-party library because we shied away from doing the right thing by them. This background is advised from my review of the state of the art, but others may have alternative evidence/commentary to present as well that could sway the discussion. It just seems like a weak argument that we should do the easy thing at the expense of doing the right thing by the application community. > > > I certainly meant this email to stimulate conversation for the teleconference tomorrow. In particular, I would like those on the list with experience building ABFT/Natural FT applications/libraries (UTK?) to express their perspective on this topic. Hopefully they can help guide us towards the right solution, which might just be a SHRINK-like mode. > > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] MPI_ANY_SOURCE ... again...
by Josh Hursey 27 Jan '12

27 Jan '12

Honestly, I do not fully understand the progress critique either. As part of my earlier email, I asked that those that understand/support that critique clarify the position so we can consider it fully. The way I currently understand the critique is that MPI_{I}recv(MPI_ANY_SOURCE) can be seen as a passive action. An implementation can locally post the receive and be passive in waiting for a matching incoming receive. So if process A posts an MPI_ANY_SOURCE receive, and process B fails. If A had never talked with B then the 'return on process failure' semantics would require that A eventually find out about the death of B in order to return with an error. I think the progress argument is that the receive operation now has to actively look around (or depend on a detector that does the looking for it) in order to complete the operation in error. The counter argument (as a few have mentioned in this thread) is that MPI implementations are already required to provide such a notification mechanism to support (at least) MPI_ERRORS_ARE_FATAL and cleanup. So if this is what they are calling progress, then the standard is already requiring it and we are just logically hooking into it. But that is my understanding of the position, which could be missing something. So I would really love to hear someone help me/us understand the critique better. Note that the progress critique was also applied to the FailHandlers which require similar completeness of failure notification. So addressing the progress critique will help both the FailHandlers and MPI_ANY_SOURCE discussions. -- Josh On Fri, Jan 27, 2012 at 3:32 PM, Fab Tillier <ftillier(a)microsoft.com> wrote: > Hi Josh, > > Any solution like (2) that blocks is fundamentally incompatible with > anything but MPI_THREAD_MULTIPLE. Requiring MPI_THREAD_MULTIPLE support in > order to support FT apps is a horrible idea. > > I don't understand the progress issue, so before I can make any arguments > in support of (1), can you summarize what the objection to (1) is? Is it > really that contentious that the failure detector would be hooked into the > MPI library and allow it to make progress by failing MPI_ANY_SOURCE > receives? Maybe that can become a property of the failure detector? > > Cheers, > -Fab > > Josh Hursey wrote on Fri, 27 Jan 2012 at 06:46:32 > > > Maybe we should take a different tact on this thread. There are clearly > > two camps here: (1) MPI_ANY_SOURCE returns on process failure, (2) > > MPI_ANY_SOURCE blocks. We are all concerned about what can and should go > > into this round of the proposal so that we have a successful first > > reading during (hopefully) the next MPI Forum meeting. So our hearts are > > all in the right place, and we should not lose sight of that. > > > > Most of the motivation for (2) is that there has been expressed concern > > about implying progress by the definition of (1). I have yet to hear > > anyone argue that (2) is better for the user, so the progress issue > > seems to be the central issue. As such, let us discuss the merits of the > > progress critique. Because if we stick with (1) we will need to have a > > well reasoned argument against this critique when it comes up again > > during the next reading. If we go with (2) then we will have to explain > > the merits of this critique to those that think (1) is the better > > choice. So either way we need a firm assessment of this issue. > > > > So would someone like to start the discussion by advocating for the > > progress critique? > > > > Maybe it would be useful to try to setup a teleconf to discuss just this > > issue - maybe trying to include those that raised concern during the > > meeting in the discussion. > > > > Thanks, > > Josh > > > > > > On Thu, Jan 26, 2012 at 4:59 PM, Graham, Richard L. <rlgraham(a)ornl.gov > > <mailto:[email protected]> > wrote: > > > > Sayantan, for some reason you seem to be against what is being proposed > > for recovery when there is failure and a posted blocking wild card > > receive. > > > > Posting to self will require threads be active, and to implement at the > > user level the same type of monitoring that is needed at the MPI level. > > I am also pretty sure that in your current MPI implementation you have > > to check (does not have to be active - it can be passive at the MPI > > level) for failed processes, so that you can terminate cleanly on > > failure, so no new requirement here. > > > > I have worked on apps that would hang (even 100% of the time) under the > > scenario you are proposing, where all their receives are wild-carded, in > > the compute portion of the code. While this is not the way we may write > > apps, this is how some are written. There are good reasons for apps to > > use wild card receives, and some users have just written codes that way > > - whether we like it or not. The subject of removing wild card receives > > has come up many times, but it does not last long, as that is simply not > > practical. > > > > Rich > > > > On Jan 26, 2012, at 4:27 PM, Sur, Sayantan wrote: > > > >> Your point is well taken. But I do not agree with your assertion that > > the situation is "unrecoverable". For example, the app/lib using > > blocking recv could use another thread to post a self send and satisfy > > that blocking receive. I know this is not elegant, but still > > recoverable, right? :-) > >> > >> I also disagree with your statement that this does not have impact on > > implementation cost. This specifically requires MPI library to have > > sustained interaction with process management subsystem to keep > > polling/watching the entire system to see where the failure occurred. It > > needs to do so at some pre-determined frequency, which may or may not be > > in the granularity the application requires. In the case where Recv(ANY) > > is posted on a sub communicator, you would also need to convey the > > participants of the sub communicator to the out-of-band system, which > > means you need to have the layout available. Another alternative is to > > send info on process failures to -everyone- and then have the MPI > > library pick the process to deliver this error to. IMHO, this is > > significant implementation complexity - not saying it can't be done > > TODAY, but what about at scale few years from now? > >> > >> This is after all a compromise - if we (app writers + MPI community) > >> can live with something less than what is ideal, should we go for it? > >> > >> Thanks. > >> > >> === > >> Sayantan Sur, Ph.D. > >> Intel Corp. > >> > >>> -----Original Message----- > >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org > > <mailto:[email protected]> [mailto:mpi3-ft- > > <mailto:mpi3-ft-> > >>> bounces(a)lists.mpi-forum.org <mailto:[email protected]> ] On > >>> Behalf Of Aurélien Bouteiller Sent: Thursday, January 26, 2012 10:40 > >>> AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >>> Group Subject: Re: [Mpi3-ft] MPI_ANY_SOURCE ... again... > >>> > >>> In my opinion, a fault tolerant proposal in which a valid MPI program > >>> deadlocks, without a remedy, in case of failure, does not address the > >>> issue, and is worthless. I do not support the proposition that ANY_TAG > >>> should just not work. -Every- function of the MPI standard should have > >>> a well defined behavior in case of failure, that do not result in the > >>> application being in a unrecoverable state. > >>> > >>> Moreover, the cost of handling any-source in inconsequent on > >>> implementation performance. The opposition are only of intellectual > >>> prettiness order; not implementation cost issues. This is not a good > >>> reason to drop functionality to the point where the proposal does not > >>> even tolerate failures of valid MPI applications anymore. > >>> > >>> > >>> Aurelien > >>> > >>> > >>> Le 26 janv. 2012 ą 11:24, Bronevetsky, Greg a écrit : > >>> > >>>> Actually, I think that we can be backwards compatible. Define > > semantics to > >>> say that we don't guarantee that MPI_ANY_SOURCE will get unblocked due > >>> to failure but don't preclude this possibility. > >>>> > >>>> I think I'm coming down on the blocks side as well but for a less > > pessimistic > >>> reason than Josh. Standards bodies are conservative for a reason: > >>> mistakes in standards are expensive. As such, if there is any feature > >>> that can be evaluated outside the standard before being included in > >>> the standard, then this is the preferable path. MPI_ANY_SOURCE returns > >>> is exactly such a feature. Sure, users will be harmed in the short > >>> term but if this is not quite the best semantics, then they'll be > >>> harmed in the long term. > >>>> > >>>> As such, lets go for the most barebones spec we can come up with on > > top > >>> of which we can implement all the other functionality we think is > >>> important. This gives us the flexibility to try out several APIs and > >>> decide on which is best before we come back before the forum to > >>> standardize the full MPI-FT API. At that point in time we'll have done > >>> a significantly stronger evaluation, which will make it much more > >>> difficult for the forum to say no, even though the list of features > >>> will be significantly more extensive in that proposal. > >>>> > >>>> Greg Bronevetsky > >>>> Lawrence Livermore National Lab > >>>> (925) 424-5756 <tel:%28925%29%20424-5756> > >>>> bronevetsky(a)llnl.gov <mailto:[email protected]> > >>>> http://greg.bronevetsky.com <http://greg.bronevetsky.com> > >>>> > >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org > >>>> <mailto:[email protected]> > >>>> [mailto:[email protected] > >>>> <mailto:[email protected]> ] On Behalf OfJosh > >>>> Hursey Sent: Thursday, January 26, 2012 8:11 AM To: MPI 3.0 Fault > >>>> Tolerance and Dynamic Process Control working Group Subject: Re: > >>>> [Mpi3-ft] MPI_ANY_SOURCE ... again... > >>>> > >>>> I mentioned something similar to folks here yesterday. If we decide > > on the > >>> "MPI_ANY_SOURCE blocks" semantics then I would expose the failure > >>> notification through an Open MPI specific API (or message in a special > >>> context, as Greg suggests), and the third-party library could short > >>> circuit their implementation of the detector/notifier with this call. > >>> I would then strongly advocate that other MPI implementations do the > >>> same. > >>>> > >>>> I think the distasteful aspect of the "MPI_ANY_SOURCE blocks" path > > is that > >>> if we are going to advocate that MPI implementations provide this type > >>> of interface anyway then why can we not just expose these semantics in > >>> the standard and be done with it. With the proposed work around, > >>> applications will be programming to the MPI standard and the semantics > >>> of this third- party library for MPI_ANY_SOURCE. So it just seems like > >>> we are working around a stalemate argument in the forum and the users > >>> end up suffering. However, if we stick with the "MPI_ANY_SOURCE > >>> returns on error" semantics we would have to thread the needle of the > >>> progress discussion (possible, but it may take quite a long time). > >>>> > >>>> So this is where I'm torn. If there was a backwards compatible path > >>> between the "MPI_ANY_SOURCE blocks" to the "MPI_ANY_SOURCE returns on > >>> error" then it would be easier to go with the former and propose the > >>> latter in a separate ticket. Then have the progress discussion over > >>> the separate ticket, and not the general RTS proposal. Maybe that path > >>> is defining a new MPI_ANY_SORCE_RETURN_ON_PROC_FAIL wildcard. > >>>> > >>>> I am, unfortunately, starting to lean towards the "MPI_ANY_SOURCE > >>> blocks" camp. I say 'unfortunately' because it hurts users, and that's > >>> not something we should be considering in my opinion... > >>>> > >>>> Good comments, keep them coming. > >>>> > >>>> -- Josh > >>>> > >>>> On Thu, Jan 26, 2012 at 10:48 AM, Bronevetsky, Greg > >>>> <bronevetsky1(a)llnl.gov <mailto:[email protected]> > wrote: I > >>>> agree that the "MPI_ANY_SOURCE returns on error" semantics is > > better > >>> for users. However, if this is going to be a sticking point for the > >>> rest of the forum, it is not actually that difficult to fake this > >>> functionality on top of the "MPI_ANY_SOURCE blocks" semantics. Josh, > >>> you correctly pointed out that the MPI implementation should be able > >>> to leverage its own out of band failure detectors to implement the > >>> "returns" functionality but if that is the case, why can't the vendor > >>> provide an optional layer to the user that will do exactly the same > >>> thing but without messing with the MPI forum? > >>>> > >>>> What I'm proposing is that vendors or any other software developers > >>> provide a failure notification layer that sends an MPI message on a > >>> pre- defined communicator to the process. They would also provide a > >>> PMPI layer that wraps MPI_Receive(MPI_ANY_SOURCE) so that it > >>> alternates between testing for the arrival of messages that match this > >>> operation and a failure notification message. If the former arrives > >>> first, the wrapper returns normally. If the latter arrives first, the > >>> original MPI_Receive(MPI_ANY_SOURCE) is cancelled and the call returns > >>> with an error. Conveniently, since the failure notifier and the PMPI > >>> layer are orthogonal, we can connect the application to any failure > >>> detector, making it possible to provide these for systems where the > >>> vendors are lazy. > >>>> > >>>> Greg Bronevetsky > >>>> Lawrence Livermore National Lab > >>>> (925) 424-5756 <tel:%28925%29%20424-5756> > >>>> bronevetsky(a)llnl.gov <mailto:[email protected]> > >>>> http://greg.bronevetsky.com <http://greg.bronevetsky.com> > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org > > <mailto:[email protected]> [mailto:mpi3-ft- > > <mailto:mpi3-ft-> > >>>>> bounces(a)lists.mpi-forum.org <mailto:[email protected]> ] > >>>>> On Behalf Of Martin Schulz Sent: Wednesday, January 25, 2012 4:30 PM > >>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >>>>> Group Subject: Re: [Mpi3-ft] MPI_ANY_SOURCE ... again... > >>>>> > >>>>> Hi Josh, all, > >>>>> > >>>>> I agree, I think the current approach is fine. Blocking is likely to > >>>>> be more problematic in many cases, IMHO. However, I am still a bit > >>>>> worried about splitting the semantics for P2P and Collective > >>>>> routines. I don't see a reason why a communicator after a collective > >>>>> call to validate wouldn't support ANY_SOURCE. If it is split, > >>>>> though, any intermediate layer trying to replace collectives with > >>>>> P2P solutions (and there are plenty of tuning frameworks out there > >>>>> that try exactly that) have a hard time maintaing the same > > semantics in an > >>> error case. > >>>>> > >>>>> Martin > >>>>> > >>>>> > >>>>> On Jan 25, 2012, at 3:06 PM, Howard Pritchard wrote: > >>>>> > >>>>>> Hi Josh, > >>>>>> > >>>>>> Cray is okay with the semantics described in the current FTWG > >>>>>> proposal attached to the ticket. > >>>>>> > >>>>>> We plan to just leverage the out-of-band system fault detector > >>>>>> software that currently kills jobs if a node goes down that the > >>>>>> job was running on. > >>>>>> > >>>>>> Howard > >>>>>> > >>>>>> Josh Hursey wrote: > >>>>>>> We really need to make a decision on semantics for MPI_ANY_SOURCE. > >>>>>>> > >>>>>>> During the plenary session the MPI Forum had a problem with the > >>>>>>> current proposed semantics. The current proposal states (roughly) > >>>>>>> that MPI_ANY_SOURCE return when a failure emerges in the > >>>>>>> communicator. The MPI Forum read this as a strong requirement for > >>>>>>> -progress- (something the MPI standard tries to stay away from). > >>>>>>> > >>>>>>> The alternative proposal is that a receive on MPI_ANY_SOURCE will > >>>>>>> block until completed with a message. This means that it will > >>>>>>> -not- return when a new failure has been encountered (even if the > >>>>>>> calling process is the only process left alive in the > >>>>>>> communicator). This does get around the concern about progress, > >>>>>>> but puts a large burden on the end user. > >>>>>>> > >>>>>>> > >>>>>>> There are a couple good use cases for MPI_ANY_SOURCE (grumble, > >>>>>>> grumble) > >>>>>>> - Manager/Worker applications, and easy load balancing when > >>>>>>> multiple incoming messages are expected. This blocking behavior > >>>>>>> makes the use of MPI_ANY_SOURCE dangerous for fault tolerant > >>>>>>> applications, and opens up another opportunity for deadlock. > >>>>>>> > >>>>>>> For applications that want to use MPI_ANY_SOURCE and be fault > >>>>>>> tolerant they will need to build their own failure detector on > >>>>>>> top of MPI using directed point-to-point messages. A basic > >>>>>>> implementation might post MPI_Irecv()'s to each worker process > >>>>>>> with an unused tag, then poll on Testany(). If any of these > >>>>>>> requests complete in error > >>>>>>> (MPI_ERR_PROC_FAIL_STOP) then the target has failed and the > >>>>>>> application can take action. This user-level failure detector can > >>>>>>> (should) be implemented in a third-party library since failure > >>>>>>> detectors can be difficult to implement in a scalable manner. > >>>>>>> > >>>>>>> In reality, the MPI library or the runtime system that supports > >>>>>>> MPI will already be doing something similar. Even for > >>>>>>> MPI_ERRORS_ARE_FATAL on MPI_COMM_WORLD, the underlying > >>> system > >>>>> must > >>>>>>> detect the process failure, and terminate all other processes in > >>>>>>> MPI_COMM_WORLD. So this represents a -detection- of the failure, > >>>>>>> and a -notification- of the failure throughout the system (though > >>>>>>> the notification is an order to terminate). For MPI_ERRORS_RETURN, > >>>>>>> the MPI will use this detection/notification functionality to > >>>>>>> reason about the state of the message traffic in the system. So it > >>>>>>> seems silly to force the user to duplicate this (nontrivial) > >>>>>>> detection/notification functionality on top of MPI, just to avoid > >>>>>>> the progress discussion. > >>>>>>> > >>>>>>> > >>>>>>> So that is a rough summary of the debate. If we are going to move > >>>>>>> forward, we need to make a decision on MPI_ANY_SOURCE. I would > >>>>>>> like to make such a decision before/during the next teleconf (Feb. > >>>>>>> 1). > >>>>>>> > >>>>>>> I'm torn on this one, so I look forward to your comments. > >>>>>>> > >>>>>>> -- Josh > >>>>>>> > >>>>>>> -- > >>>>>>> Joshua Hursey > >>>>>>> Postdoctoral Research Associate > >>>>>>> Oak Ridge National Laboratory > >>>>>>> http://users.nccs.gov/~jjhursey <http://users.nccs.gov/~jjhursey> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Howard Pritchard > >>>>>> Software Engineering > >>>>>> Cray, Inc. > >>>>>> _______________________________________________ > >>>>>> mpi3-ft mailing list > >>>>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> > >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > >>>>> > >>>>> > >>> __________________________________________________________ > >>> ____ > >>>>> __________ Martin Schulz, schulzm(a)llnl.gov <mailto:[email protected]> > >>>>> , http://people.llnl.gov/schulzm <http://people.llnl.gov/schulzm> > >>>>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> mpi3-ft mailing list > >>>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> > >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > >>>> > >>>> _______________________________________________ mpi3-ft mailing list > >>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >>>> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Joshua Hursey > >>>> Postdoctoral Research Associate > >>>> Oak Ridge National Laboratory > >>>> http://users.nccs.gov/~jjhursey <http://users.nccs.gov/~jjhursey> > >>>> _______________________________________________ > >>>> mpi3-ft mailing list > >>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > >>> > >>> -- > >>> * Dr. Aurélien Bouteiller > >>> * Researcher at Innovative Computing Laboratory > >>> * University of Tennessee > >>> * 1122 Volunteer Boulevard, suite 350 > >>> * Knoxville, TN 37996 > >>> * 865 974 6321 <tel:865%20974%206321> > >>> > >>> > >>> > >>> > >> > >> > >> _______________________________________________ > >> mpi3-ft mailing list > >> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > > > > > > _______________________________________________ mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > > > > > > > > > > -- > > Joshua Hursey > > Postdoctoral Research Associate > > Oak Ridge National Laboratory > > http://users.nccs.gov/~jjhursey <http://users.nccs.gov/~jjhursey> > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_ANY_SOURCE ... again...
by Graham, Richard L. 27 Jan '12

27 Jan '12

How is the failure detector needed for FT any different than failure detectors we use today to ensure proper cleanup ? There must be additional data sent around that is not today, but I don't see a fundamental difference. Rich On Jan 27, 2012, at 3:32 PM, Fab Tillier wrote: > Hi Josh, > > Any solution like (2) that blocks is fundamentally incompatible with anything but MPI_THREAD_MULTIPLE. Requiring MPI_THREAD_MULTIPLE support in order to support FT apps is a horrible idea. > > I don't understand the progress issue, so before I can make any arguments in support of (1), can you summarize what the objection to (1) is? Is it really that contentious that the failure detector would be hooked into the MPI library and allow it to make progress by failing MPI_ANY_SOURCE receives? Maybe that can become a property of the failure detector? > > Cheers, > -Fab > > Josh Hursey wrote on Fri, 27 Jan 2012 at 06:46:32 > >> Maybe we should take a different tact on this thread. There are clearly >> two camps here: (1) MPI_ANY_SOURCE returns on process failure, (2) >> MPI_ANY_SOURCE blocks. We are all concerned about what can and should go >> into this round of the proposal so that we have a successful first >> reading during (hopefully) the next MPI Forum meeting. So our hearts are >> all in the right place, and we should not lose sight of that. >> >> Most of the motivation for (2) is that there has been expressed concern >> about implying progress by the definition of (1). I have yet to hear >> anyone argue that (2) is better for the user, so the progress issue >> seems to be the central issue. As such, let us discuss the merits of the >> progress critique. Because if we stick with (1) we will need to have a >> well reasoned argument against this critique when it comes up again >> during the next reading. If we go with (2) then we will have to explain >> the merits of this critique to those that think (1) is the better >> choice. So either way we need a firm assessment of this issue. >> >> So would someone like to start the discussion by advocating for the >> progress critique? >> >> Maybe it would be useful to try to setup a teleconf to discuss just this >> issue - maybe trying to include those that raised concern during the >> meeting in the discussion. >> >> Thanks, >> Josh >> >> >> On Thu, Jan 26, 2012 at 4:59 PM, Graham, Richard L. <rlgraham(a)ornl.gov >> <mailto:[email protected]> > wrote: >> >> Sayantan, for some reason you seem to be against what is being proposed >> for recovery when there is failure and a posted blocking wild card >> receive. >> >> Posting to self will require threads be active, and to implement at the >> user level the same type of monitoring that is needed at the MPI level. >> I am also pretty sure that in your current MPI implementation you have >> to check (does not have to be active - it can be passive at the MPI >> level) for failed processes, so that you can terminate cleanly on >> failure, so no new requirement here. >> >> I have worked on apps that would hang (even 100% of the time) under the >> scenario you are proposing, where all their receives are wild-carded, in >> the compute portion of the code. While this is not the way we may write >> apps, this is how some are written. There are good reasons for apps to >> use wild card receives, and some users have just written codes that way >> - whether we like it or not. The subject of removing wild card receives >> has come up many times, but it does not last long, as that is simply not >> practical. >> >> Rich >> >> On Jan 26, 2012, at 4:27 PM, Sur, Sayantan wrote: >> >>> Your point is well taken. But I do not agree with your assertion that >> the situation is "unrecoverable". For example, the app/lib using >> blocking recv could use another thread to post a self send and satisfy >> that blocking receive. I know this is not elegant, but still >> recoverable, right? :-) >>> >>> I also disagree with your statement that this does not have impact on >> implementation cost. This specifically requires MPI library to have >> sustained interaction with process management subsystem to keep >> polling/watching the entire system to see where the failure occurred. It >> needs to do so at some pre-determined frequency, which may or may not be >> in the granularity the application requires. In the case where Recv(ANY) >> is posted on a sub communicator, you would also need to convey the >> participants of the sub communicator to the out-of-band system, which >> means you need to have the layout available. Another alternative is to >> send info on process failures to -everyone- and then have the MPI >> library pick the process to deliver this error to. IMHO, this is >> significant implementation complexity - not saying it can't be done >> TODAY, but what about at scale few years from now? >>> >>> This is after all a compromise - if we (app writers + MPI community) >>> can live with something less than what is ideal, should we go for it? >>> >>> Thanks. >>> >>> === >>> Sayantan Sur, Ph.D. >>> Intel Corp. >>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org >> <mailto:[email protected]> [mailto:mpi3-ft- >> <mailto:mpi3-ft-> >>>> bounces(a)lists.mpi-forum.org <mailto:[email protected]> ] On >>>> Behalf Of Aurélien Bouteiller Sent: Thursday, January 26, 2012 10:40 >>>> AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>>> Group Subject: Re: [Mpi3-ft] MPI_ANY_SOURCE ... again... >>>> >>>> In my opinion, a fault tolerant proposal in which a valid MPI program >>>> deadlocks, without a remedy, in case of failure, does not address the >>>> issue, and is worthless. I do not support the proposition that ANY_TAG >>>> should just not work. -Every- function of the MPI standard should have >>>> a well defined behavior in case of failure, that do not result in the >>>> application being in a unrecoverable state. >>>> >>>> Moreover, the cost of handling any-source in inconsequent on >>>> implementation performance. The opposition are only of intellectual >>>> prettiness order; not implementation cost issues. This is not a good >>>> reason to drop functionality to the point where the proposal does not >>>> even tolerate failures of valid MPI applications anymore. >>>> >>>> >>>> Aurelien >>>> >>>> >>>> Le 26 janv. 2012 ą 11:24, Bronevetsky, Greg a écrit : >>>> >>>>> Actually, I think that we can be backwards compatible. Define >> semantics to >>>> say that we don't guarantee that MPI_ANY_SOURCE will get unblocked due >>>> to failure but don't preclude this possibility. >>>>> >>>>> I think I'm coming down on the blocks side as well but for a less >> pessimistic >>>> reason than Josh. Standards bodies are conservative for a reason: >>>> mistakes in standards are expensive. As such, if there is any feature >>>> that can be evaluated outside the standard before being included in >>>> the standard, then this is the preferable path. MPI_ANY_SOURCE returns >>>> is exactly such a feature. Sure, users will be harmed in the short >>>> term but if this is not quite the best semantics, then they'll be >>>> harmed in the long term. >>>>> >>>>> As such, lets go for the most barebones spec we can come up with on >> top >>>> of which we can implement all the other functionality we think is >>>> important. This gives us the flexibility to try out several APIs and >>>> decide on which is best before we come back before the forum to >>>> standardize the full MPI-FT API. At that point in time we'll have done >>>> a significantly stronger evaluation, which will make it much more >>>> difficult for the forum to say no, even though the list of features >>>> will be significantly more extensive in that proposal. >>>>> >>>>> Greg Bronevetsky >>>>> Lawrence Livermore National Lab >>>>> (925) 424-5756 <tel:%28925%29%20424-5756> >>>>> bronevetsky(a)llnl.gov <mailto:[email protected]> >>>>> http://greg.bronevetsky.com <http://greg.bronevetsky.com> >>>>> >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org >>>>> <mailto:[email protected]> >>>>> [mailto:[email protected] >>>>> <mailto:[email protected]> ] On Behalf OfJosh >>>>> Hursey Sent: Thursday, January 26, 2012 8:11 AM To: MPI 3.0 Fault >>>>> Tolerance and Dynamic Process Control working Group Subject: Re: >>>>> [Mpi3-ft] MPI_ANY_SOURCE ... again... >>>>> >>>>> I mentioned something similar to folks here yesterday. If we decide >> on the >>>> "MPI_ANY_SOURCE blocks" semantics then I would expose the failure >>>> notification through an Open MPI specific API (or message in a special >>>> context, as Greg suggests), and the third-party library could short >>>> circuit their implementation of the detector/notifier with this call. >>>> I would then strongly advocate that other MPI implementations do the >>>> same. >>>>> >>>>> I think the distasteful aspect of the "MPI_ANY_SOURCE blocks" path >> is that >>>> if we are going to advocate that MPI implementations provide this type >>>> of interface anyway then why can we not just expose these semantics in >>>> the standard and be done with it. With the proposed work around, >>>> applications will be programming to the MPI standard and the semantics >>>> of this third- party library for MPI_ANY_SOURCE. So it just seems like >>>> we are working around a stalemate argument in the forum and the users >>>> end up suffering. However, if we stick with the "MPI_ANY_SOURCE >>>> returns on error" semantics we would have to thread the needle of the >>>> progress discussion (possible, but it may take quite a long time). >>>>> >>>>> So this is where I'm torn. If there was a backwards compatible path >>>> between the "MPI_ANY_SOURCE blocks" to the "MPI_ANY_SOURCE returns on >>>> error" then it would be easier to go with the former and propose the >>>> latter in a separate ticket. Then have the progress discussion over >>>> the separate ticket, and not the general RTS proposal. Maybe that path >>>> is defining a new MPI_ANY_SORCE_RETURN_ON_PROC_FAIL wildcard. >>>>> >>>>> I am, unfortunately, starting to lean towards the "MPI_ANY_SOURCE >>>> blocks" camp. I say 'unfortunately' because it hurts users, and that's >>>> not something we should be considering in my opinion... >>>>> >>>>> Good comments, keep them coming. >>>>> >>>>> -- Josh >>>>> >>>>> On Thu, Jan 26, 2012 at 10:48 AM, Bronevetsky, Greg >>>>> <bronevetsky1(a)llnl.gov <mailto:[email protected]> > wrote: I >>>>> agree that the "MPI_ANY_SOURCE returns on error" semantics is >> better >>>> for users. However, if this is going to be a sticking point for the >>>> rest of the forum, it is not actually that difficult to fake this >>>> functionality on top of the "MPI_ANY_SOURCE blocks" semantics. Josh, >>>> you correctly pointed out that the MPI implementation should be able >>>> to leverage its own out of band failure detectors to implement the >>>> "returns" functionality but if that is the case, why can't the vendor >>>> provide an optional layer to the user that will do exactly the same >>>> thing but without messing with the MPI forum? >>>>> >>>>> What I'm proposing is that vendors or any other software developers >>>> provide a failure notification layer that sends an MPI message on a >>>> pre- defined communicator to the process. They would also provide a >>>> PMPI layer that wraps MPI_Receive(MPI_ANY_SOURCE) so that it >>>> alternates between testing for the arrival of messages that match this >>>> operation and a failure notification message. If the former arrives >>>> first, the wrapper returns normally. If the latter arrives first, the >>>> original MPI_Receive(MPI_ANY_SOURCE) is cancelled and the call returns >>>> with an error. Conveniently, since the failure notifier and the PMPI >>>> layer are orthogonal, we can connect the application to any failure >>>> detector, making it possible to provide these for systems where the >>>> vendors are lazy. >>>>> >>>>> Greg Bronevetsky >>>>> Lawrence Livermore National Lab >>>>> (925) 424-5756 <tel:%28925%29%20424-5756> >>>>> bronevetsky(a)llnl.gov <mailto:[email protected]> >>>>> http://greg.bronevetsky.com <http://greg.bronevetsky.com> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org >> <mailto:[email protected]> [mailto:mpi3-ft- >> <mailto:mpi3-ft-> >>>>>> bounces(a)lists.mpi-forum.org <mailto:[email protected]> ] >>>>>> On Behalf Of Martin Schulz Sent: Wednesday, January 25, 2012 4:30 PM >>>>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>>>>> Group Subject: Re: [Mpi3-ft] MPI_ANY_SOURCE ... again... >>>>>> >>>>>> Hi Josh, all, >>>>>> >>>>>> I agree, I think the current approach is fine. Blocking is likely to >>>>>> be more problematic in many cases, IMHO. However, I am still a bit >>>>>> worried about splitting the semantics for P2P and Collective >>>>>> routines. I don't see a reason why a communicator after a collective >>>>>> call to validate wouldn't support ANY_SOURCE. If it is split, >>>>>> though, any intermediate layer trying to replace collectives with >>>>>> P2P solutions (and there are plenty of tuning frameworks out there >>>>>> that try exactly that) have a hard time maintaing the same >> semantics in an >>>> error case. >>>>>> >>>>>> Martin >>>>>> >>>>>> >>>>>> On Jan 25, 2012, at 3:06 PM, Howard Pritchard wrote: >>>>>> >>>>>>> Hi Josh, >>>>>>> >>>>>>> Cray is okay with the semantics described in the current FTWG >>>>>>> proposal attached to the ticket. >>>>>>> >>>>>>> We plan to just leverage the out-of-band system fault detector >>>>>>> software that currently kills jobs if a node goes down that the >>>>>>> job was running on. >>>>>>> >>>>>>> Howard >>>>>>> >>>>>>> Josh Hursey wrote: >>>>>>>> We really need to make a decision on semantics for MPI_ANY_SOURCE. >>>>>>>> >>>>>>>> During the plenary session the MPI Forum had a problem with the >>>>>>>> current proposed semantics. The current proposal states (roughly) >>>>>>>> that MPI_ANY_SOURCE return when a failure emerges in the >>>>>>>> communicator. The MPI Forum read this as a strong requirement for >>>>>>>> -progress- (something the MPI standard tries to stay away from). >>>>>>>> >>>>>>>> The alternative proposal is that a receive on MPI_ANY_SOURCE will >>>>>>>> block until completed with a message. This means that it will >>>>>>>> -not- return when a new failure has been encountered (even if the >>>>>>>> calling process is the only process left alive in the >>>>>>>> communicator). This does get around the concern about progress, >>>>>>>> but puts a large burden on the end user. >>>>>>>> >>>>>>>> >>>>>>>> There are a couple good use cases for MPI_ANY_SOURCE (grumble, >>>>>>>> grumble) >>>>>>>> - Manager/Worker applications, and easy load balancing when >>>>>>>> multiple incoming messages are expected. This blocking behavior >>>>>>>> makes the use of MPI_ANY_SOURCE dangerous for fault tolerant >>>>>>>> applications, and opens up another opportunity for deadlock. >>>>>>>> >>>>>>>> For applications that want to use MPI_ANY_SOURCE and be fault >>>>>>>> tolerant they will need to build their own failure detector on >>>>>>>> top of MPI using directed point-to-point messages. A basic >>>>>>>> implementation might post MPI_Irecv()'s to each worker process >>>>>>>> with an unused tag, then poll on Testany(). If any of these >>>>>>>> requests complete in error >>>>>>>> (MPI_ERR_PROC_FAIL_STOP) then the target has failed and the >>>>>>>> application can take action. This user-level failure detector can >>>>>>>> (should) be implemented in a third-party library since failure >>>>>>>> detectors can be difficult to implement in a scalable manner. >>>>>>>> >>>>>>>> In reality, the MPI library or the runtime system that supports >>>>>>>> MPI will already be doing something similar. Even for >>>>>>>> MPI_ERRORS_ARE_FATAL on MPI_COMM_WORLD, the underlying >>>> system >>>>>> must >>>>>>>> detect the process failure, and terminate all other processes in >>>>>>>> MPI_COMM_WORLD. So this represents a -detection- of the failure, >>>>>>>> and a -notification- of the failure throughout the system (though >>>>>>>> the notification is an order to terminate). For MPI_ERRORS_RETURN, >>>>>>>> the MPI will use this detection/notification functionality to >>>>>>>> reason about the state of the message traffic in the system. So it >>>>>>>> seems silly to force the user to duplicate this (nontrivial) >>>>>>>> detection/notification functionality on top of MPI, just to avoid >>>>>>>> the progress discussion. >>>>>>>> >>>>>>>> >>>>>>>> So that is a rough summary of the debate. If we are going to move >>>>>>>> forward, we need to make a decision on MPI_ANY_SOURCE. I would >>>>>>>> like to make such a decision before/during the next teleconf (Feb. >>>>>>>> 1). >>>>>>>> >>>>>>>> I'm torn on this one, so I look forward to your comments. >>>>>>>> >>>>>>>> -- Josh >>>>>>>> >>>>>>>> -- >>>>>>>> Joshua Hursey >>>>>>>> Postdoctoral Research Associate >>>>>>>> Oak Ridge National Laboratory >>>>>>>> http://users.nccs.gov/~jjhursey <http://users.nccs.gov/~jjhursey> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Howard Pritchard >>>>>>> Software Engineering >>>>>>> Cray, Inc. >>>>>>> _______________________________________________ >>>>>>> mpi3-ft mailing list >>>>>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >>>>>> >>>>>> >>>> __________________________________________________________ >>>> ____ >>>>>> __________ Martin Schulz, schulzm(a)llnl.gov <mailto:[email protected]> >>>>>> , http://people.llnl.gov/schulzm <http://people.llnl.gov/schulzm> >>>>>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >>>>> >>>>> _______________________________________________ mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Joshua Hursey >>>>> Postdoctoral Research Associate >>>>> Oak Ridge National Laboratory >>>>> http://users.nccs.gov/~jjhursey <http://users.nccs.gov/~jjhursey> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >>>> >>>> -- >>>> * Dr. Aurélien Bouteiller >>>> * Researcher at Innovative Computing Laboratory >>>> * University of Tennessee >>>> * 1122 Volunteer Boulevard, suite 350 >>>> * Knoxville, TN 37996 >>>> * 865 974 6321 <tel:865%20974%206321> >>>> >>>> >>>> >>>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >> >> >> _______________________________________________ mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org <mailto:[email protected]> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> <http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey <http://users.nccs.gov/~jjhursey> >> > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0