February 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Fault Tolerance & RMA Discussion
by Josh Hursey 07 Feb '12

07 Feb '12

Attached are my notes from the RMA meeting this morning. Thanks, Josh On Mon, Feb 6, 2012 at 2:45 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > We are going to meet from 10-11 am (Eastern) on Feb. 7 to continue our > conversation. We will use the same call-in information as before. > > Thanks, > Josh > > > On Thu, Feb 2, 2012 at 3:00 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > >> We made some really good progress on today's call. Attached are some >> notes that I took from the call. >> >> At the end of the call there were a couple of items that we wanted to get >> a finer understanding of. As a result we are going to try to setup another >> teleconf. >> >> Below is a doodle poll to pick a date/time: >> http://www.doodle.com/kzmiknie8yz4wxkc >> >> If you are interested in attending this teleconf, please fill out the >> poll by 2 pm Eastern on Monday, Feb. 6. >> >> Thanks, >> Josh >> >> >> On Thu, Feb 2, 2012 at 10:01 AM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: >> >>> Just a reminder that we are meeting today at Noon Eastern to discuss RMA >>> in the context of the fault tolerance proposal. >>> >>> The Run-Through Stabilization proposal can be found attached to the >>> ticket: >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 >>> >>> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… >>> >>> We will be focusing on section 17.11 of that document. Note that this >>> section does not currently explicitly account for the new RMA proposal, but >>> we would like to remedy that for the next reading. >>> >>> Thanks, >>> Josh >>> >>> On Wed, Jan 25, 2012 at 3:15 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: >>> >>>> There was no one date/time that worked for everyone, but I chose a time >>>> that worked for most of the respondents. We will meet Thursday, Feb. 2 from >>>> 12-1 pm EST/New York to discuss this topic. >>>> >>>> We can use the following teleconf information: >>>> US Toll Free number: 877-801-8130 >>>> Toll number: 1-203-692-8690 >>>> Access Code: 1044056 >>>> >>>> Thanks, >>>> Josh >>>> >>>> >>>> On Mon, Jan 23, 2012 at 4:33 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: >>>> >>>>> (Cross posted to both the RMA and FT MPI-3 listservs) >>>>> >>>>> During the FT plenary session at the Jan. MPI Forum meeting it was >>>>> recommended that some of the members of the FT group and the RMA group have >>>>> a meeting to hash out the precise details of the FT semantics for the RMA >>>>> chapter. So I would like to facilitate such a discussion, preferability in >>>>> the next week (so we have time to fine tune things before the next forum >>>>> meeting). >>>>> >>>>> In general, we are trying to answer the question "How should RMA >>>>> operations behave when a process failure occurs?" The feeling seemed to be >>>>> that the current approach is ok (invalidating the window, forcing >>>>> recreation/validation), but the statement that the memory exposed in the >>>>> window is 'undefined' seemed excessive. The suggestion was to change the >>>>> wording to something like "Only the memory associated with a window that >>>>> was targeted by an operation that modified it is undefined after process >>>>> failure in the group associated with the window." This lead to a >>>>> considerable amount of debate in the meeting, so it was suggested that we >>>>> take the discussion offline. >>>>> >>>>> Below is a link to a doodle poll to find a good time for a teleconf. >>>>> If you are interested in participating in this discussion, please fill this >>>>> poll out by 2 PM Eastern on Wed. Jan 25 so we can set the date/time. >>>>> http://www.doodle.com/vd33va5h8iankega >>>>> >>>>> Thanks, >>>>> Josh >>>>> >>>>> -- >>>>> Joshua Hursey >>>>> Postdoctoral Research Associate >>>>> Oak Ridge National Laboratory >>>>> http://users.nccs.gov/~jjhursey >>>>> >>>> >>>> >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> >>> >>> >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Meeting on Friday, Feb. 10 (Possible Time Change)
by Josh Hursey 07 Feb '12

07 Feb '12

A doodle it is: http://www.doodle.com/5hvg59zvq84ggtfc Please fill it out by 9 am Eastern, Wednesday, Feb. 8 so we can set the time. Thanks, Josh On Tue, Feb 7, 2012 at 2:01 PM, Anthony Skjellum <tony(a)cis.uab.edu> wrote: > Doodle please > > Anthony Skjellum, PhD > 205-807-4968 > > > On Feb 7, 2012, at 12:22 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > > On the call today it was requested that we move the time of the meeting on > Friday, Feb. 10 to some time other than Noon Eastern. I'm open all day. > > It was suggested that we try for either 11 am or 1 pm (Eastern). What time > would work best for folks? > > I can start a doodle poll if we want, but I suspect we can figure it out > over email faster. > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Meeting on Friday, Feb. 10 (Possible Time Change)
by Anthony Skjellum 07 Feb '12

07 Feb '12

Doodle please Anthony Skjellum, PhD 205-807-4968 On Feb 7, 2012, at 12:22 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > On the call today it was requested that we move the time of the meeting on Friday, Feb. 10 to some time other than Noon Eastern. I'm open all day. > > It was suggested that we try for either 11 am or 1 pm (Eastern). What time would work best for folks? > > I can start a doodle poll if we want, but I suspect we can figure it out over email faster. > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Meeting on Friday, Feb. 10 (Possible Time Change)
by Sayantan Sur 07 Feb '12

07 Feb '12

I would vote for 1PM ... as 11AM EST is a little early for me. Thanks, Sayantan. On Tue, Feb 7, 2012 at 10:50 AM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > Either tome works for me. > > -d > > On Feb 7, 2012, at 11:56 AM, Josh Hursey wrote: > >> On the call today it was requested that we move the time of the meeting on Friday, Feb. 10 to some time other than Noon Eastern. I'm open all day. >> >> It was suggested that we try for either 11 am or 1 pm (Eastern). What time would work best for folks? >> >> I can start a doodle poll if we want, but I suspect we can figure it out over email faster. >> >> -- Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Meeting on Friday, Feb. 10 (Possible Time Change)
by Darius Buntinas 07 Feb '12

07 Feb '12

Either tome works for me. -d On Feb 7, 2012, at 11:56 AM, Josh Hursey wrote: > On the call today it was requested that we move the time of the meeting on Friday, Feb. 10 to some time other than Noon Eastern. I'm open all day. > > It was suggested that we try for either 11 am or 1 pm (Eastern). What time would work best for folks? > > I can start a doodle poll if we want, but I suspect we can figure it out over email faster. > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Meeting on Friday, Feb. 10 (Possible Time Change)
by Josh Hursey 07 Feb '12

07 Feb '12

On the call today it was requested that we move the time of the meeting on Friday, Feb. 10 to some time other than Noon Eastern. I'm open all day. It was suggested that we try for either 11 am or 1 pm (Eastern). What time would work best for folks? I can start a doodle poll if we want, but I suspect we can figure it out over email faster. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 07 Feb '12

07 Feb '12

Copy/Paste error. We are meeting today, Tuesday Feb 7, at Noon EST. -- Josh On Tue, Feb 7, 2012 at 9:03 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Reminder that we have a teleconf today, Wednesday, at Noon EST. > > Date: February 7, 2012 > Time: Noon EST/New York > Dial-in information: 877-801-8130 > Code: 1044056 > > > Agenda: > - Misc. > - I/O Teleconf (TBD) > - Doodle Poll: http://www.doodle.com/zw83eht6hh8mwmic > - Timeline > - March 6 - MPI Forum Meeting (first reading) > - Feb. 20 - Revised proposal due (-2 weeks) > - Feb. 13 - Draft due for working group review (-3 weeks) > - Feb. 6 - Firm decisions on pending issues (-4 weeks) > - Piggybacking (Martin S.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - Run-Through Stabilization Proposal > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… > http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php > - Review RMA teleconf discussions > - Review I/O teleconf discussions > - Decide on FailHandlers > - Decide on communicator creation matching > - Decide on Collectives over communicators with holes > - Decide on MPI_ANY_SOURCE semantic > - Other items? > > > Additional Agenda Items [as time permits]: > - FA-MPI Alternative Proposal (Tony) > http://lists.mpi-forum.org/mpi3-ft/2012/02/1048.php > - UTK-FT-MPI Alternative Proposal (UTK) > http://lists.mpi-forum.org/mpi3-ft/2012/02/1047.php > - Process Recovery (Josh H.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 > - MPI Forum notes discussion (?) > http://lists.mpi-forum.org/mpi3-ft/2012/01/0983.php > http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php > - Error to Fault changes proposal (Darius B.) > http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php > - Network Channel Failure > http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php > - FT Query Interface > http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php > > > Next Meetings: > * Feb. 7, 2012 (Teleconf) > * Feb. 10, 2012 (Teleconf) > * Feb. 15, 2012 (Teleconf) > * Feb. 22, 2012 (Teleconf - As needed) > * Feb. 29, 2012 (Teleconf) > * March 5, 2012 (MPI Forum Meeting) > * March 14, 2012 (Teleconf) > > Respond to the list if you have any additional topics that you would like > to discuss. > > -- Josh > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Alternative approach/proposals
by Ralph Castain 07 Feb '12

07 Feb '12

Hi folks I'm a newbie to the group, so please excuse my ignorance over prior discussions. I have been reading the wiki documents with interest, and have scanned the latest alternative proposals. These are some initial impressions. Having worked fault tolerance issues (albeit outside the MPI layer) for the past decade, it seems to me that the group is somewhat confronting the issue that FT techniques aren't as clear cut as we might desire - i.e., there are still multiple ways of dealing with faults, and no clear cut single approach out-shines all others. Thus, it would seem to me that taking a minimalist approach to the standard, as put forward by both alternative proposals, represents the best current next step while leaving researchers free to continue exploring the field. What I'm wondering is if the group might receive a more positive response from the Forum in general if it took a multi-step approach, not trying to fully standardize FT from the very start. I know I'm personally concerned about the probable race conditions and performance impacts of trying to fully resolve FT - having fought some of those battles over the years, I suspect we are a some distance away from cleanly handling all scenarios, especially those involving multiple, near-simultaneous failures. Perhaps we could start by proposing adoption of a standard calling for fault notification, possibly allowing that feature to be configured out so that users not requiring FT are unaffected by it? I note that both alternative proposals put forward tonight start with this basic capability, and it makes the most sense to me. If we could make MPI communication the equivalent of the more common "socket" IPC - i.e., if apps could be notified that a message failed to be delivered ala what we get from a typical socket - then that would seem to be a major step forward. The ability to not block in a procedure and allow an app to determine how to recover (perhaps providing a few simple tools as proposed by UTK) seems like something we can reasonably hope to implement without destabilizing the MPI code base. As for my suggestion that the standard stipulate that implementations are allowed to enable such things via configuration, if desired, my rationale is simply that the fault frequency we see in industry is very low as our clusters are relatively small. In talking with a few compatriots and customers, and reviewing literature over the last few weeks, it appears that clusters of even a few thousand nodes are only seeing node failures (as opposed to the more common file system errors) every few days to weeks, and that rapid checkpoint/restart capabilities are under development to render those manageable. The most widely used clusters (in terms of percent of the market) are in the 64-128 node range, and report node failures less than once a year after initial burn-in. Thus, while a fascinating research topic and one that will likely become more applicable to future "extreme scale" clusters, it seems likely that the very large majority of MPI systems do not require these capabilities. It would therefore seem more likely to be acceptable to the broader community if we left it as optional in some fashion - i.e., it should be in a "quality" implementation, but not required to be "on" unless specifically built that way. This would provide for a broader research and "try out" capability, without creating negative reactions on the part of those not requiring it. HTH a bit, and I look forward to participating more in the discussions. Ralph

1 0

Re: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 07 Feb '12

07 Feb '12

I forgot to mention. We are having another teleconf on Friday. Two new proposals were sent out this morning, we will discuss those on Friday so that folks have enough time to review to materials attached to those emails. The teleconf today will focus on the RTS proposal. Thanks, Josh On Tue, Feb 7, 2012 at 9:03 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Reminder that we have a teleconf today, Wednesday, at Noon EST. > > Date: February 7, 2012 > Time: Noon EST/New York > Dial-in information: 877-801-8130 > Code: 1044056 > > > Agenda: > - Misc. > - I/O Teleconf (TBD) > - Doodle Poll: http://www.doodle.com/zw83eht6hh8mwmic > - Timeline > - March 6 - MPI Forum Meeting (first reading) > - Feb. 20 - Revised proposal due (-2 weeks) > - Feb. 13 - Draft due for working group review (-3 weeks) > - Feb. 6 - Firm decisions on pending issues (-4 weeks) > - Piggybacking (Martin S.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - Run-Through Stabilization Proposal > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… > http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php > - Review RMA teleconf discussions > - Review I/O teleconf discussions > - Decide on FailHandlers > - Decide on communicator creation matching > - Decide on Collectives over communicators with holes > - Decide on MPI_ANY_SOURCE semantic > - Other items? > > > Additional Agenda Items [as time permits]: > - FA-MPI Alternative Proposal (Tony) > http://lists.mpi-forum.org/mpi3-ft/2012/02/1048.php > - UTK-FT-MPI Alternative Proposal (UTK) > http://lists.mpi-forum.org/mpi3-ft/2012/02/1047.php > - Process Recovery (Josh H.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 > - MPI Forum notes discussion (?) > http://lists.mpi-forum.org/mpi3-ft/2012/01/0983.php > http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php > - Error to Fault changes proposal (Darius B.) > http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php > - Network Channel Failure > http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php > - FT Query Interface > http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php > > > Next Meetings: > * Feb. 7, 2012 (Teleconf) > * Feb. 10, 2012 (Teleconf) > * Feb. 15, 2012 (Teleconf) > * Feb. 22, 2012 (Teleconf - As needed) > * Feb. 29, 2012 (Teleconf) > * March 5, 2012 (MPI Forum Meeting) > * March 14, 2012 (Teleconf) > > Respond to the list if you have any additional topics that you would like > to discuss. > > -- Josh > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 07 Feb '12

07 Feb '12

Reminder that we have a teleconf today, Wednesday, at Noon EST. Date: February 7, 2012 Time: Noon EST/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Misc. - I/O Teleconf (TBD) - Doodle Poll: http://www.doodle.com/zw83eht6hh8mwmic - Timeline - March 6 - MPI Forum Meeting (first reading) - Feb. 20 - Revised proposal due (-2 weeks) - Feb. 13 - Draft due for working group review (-3 weeks) - Feb. 6 - Firm decisions on pending issues (-4 weeks) - Piggybacking (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php - Review RMA teleconf discussions - Review I/O teleconf discussions - Decide on FailHandlers - Decide on communicator creation matching - Decide on Collectives over communicators with holes - Decide on MPI_ANY_SOURCE semantic - Other items? Additional Agenda Items [as time permits]: - FA-MPI Alternative Proposal (Tony) http://lists.mpi-forum.org/mpi3-ft/2012/02/1048.php - UTK-FT-MPI Alternative Proposal (UTK) http://lists.mpi-forum.org/mpi3-ft/2012/02/1047.php - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2012/01/0983.php http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Feb. 7, 2012 (Teleconf) * Feb. 10, 2012 (Teleconf) * Feb. 15, 2012 (Teleconf) * Feb. 22, 2012 (Teleconf - As needed) * Feb. 29, 2012 (Teleconf) * March 5, 2012 (MPI Forum Meeting) * March 14, 2012 (Teleconf) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0