February 2012 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] UAB Alternative Fault-Tolerant Proposal
by Anthony Skjellum 06 Feb '12

06 Feb '12

Dear group, As promised right after the last MPI-3 meeting, we have prepared an alternative Fault-tolerant proposal for MPI-3. This started from the premise that by starting from a different set of assumptions, we could provide a complete alternative. This starts from scratch per se, as mentioned earlier. We denote this FA-MPI or Fault-Aware MPI, to distinguish it from the main line of discussion, which is of course Fault-Tolerant MPI. The main idea is hardened collective transactions on all non-blocking MPI-3 programs. The use of all non-blocking MPI operations is designed to connect efficiently with the expectation of highly non-blocking programs for exascale, ones that emphasize overlap of communication, computation, and I/O. This is not yet condensed into an MPI-3 Format proposal (we will only do so if there is significant interest in pursuing this proposal). This lacks section 7.7 MPI-3 1-sided, 7.8 I/O, 7.9 initialization/deinitialization of I/O & 1-sided (ancillary); Nonetheless, we wanted to share the proposal now. We are still evaluating MPI-3 1-sided :-) frankly. We will be done with the File I/O in a few days; and the 1-sided shortly thereafter. We recognize that this is a quite different approach, and may or may not be appropriate for the Forum at this time. Once we get feedback, we intend to implement this model ASAP... we are just beginning implementation plans. If it does not get to be part of an MPI-3 standard, we are still interested in fully expressing this model as extensions for MPI-3. With kind regards, Tony PS Please note that this is a draft to the MPI-3 Forum, it is not a publication, just a draft. We will provide a version of this a) as an MPI-3 proposal if that's of interest, and b) this document as a finalized tech report, and later c) a conference paper :-) -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

[Mpi3-ft] An alternative FT proposal
by George Bosilca 06 Feb '12

06 Feb '12

FT working group, As announced, we have been working on an alternative FT proposal. The leading idea of this proposal is to relieve the burden of consistency from the MPI implementations, while providing the means for the user to regain control. As suggested in the mailing list a few days ago, this proposal tries to minimize the semantic changes and additional functions. We believe the proposed set of functions is minimal, yet sufficient to implement stronger consistency models, such as the current working group proposal. The proposal is not yet complete, we are still working on intercoms, RMA and file operations, but we wanted to submit it for feedback and discussion on the call this week. The UTK team. PS: The abstract and the proposal are attached below. Abstract: In this document we propose a flexible approach providing fail-stop process fault tolerance by allowing the application to react to failures while maintaining a minimal execution path in failure-free executions. Our proposal focuses on returning control to the application by avoiding deadlocks due to failures within the MPI library. No implicit, asynchronous error notification is required. Instead, functions are provided to allow processes to invalidate any communication object, thus preventing any process from waiting indefinitely on calls involving the invalidated objects. We consider the proposed set of functions to constitute a minimal basis which allows libraries and applications to increase the fault tolerance capabilities by supporting additional types of failures, and to build other desired strategies and consistency models to tolerate faults. George Bosilca Research Assistant Professor Innovative Computing Laboratory Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville http://web.eecs.utk.edu/~bosilca/

1 0

[Mpi3-ft] CFP: 2nd International Workshop on Fault-Tolerance for HPC at Extreme Scale (FTXS 2012)
by Debardeleben, Nathan A 06 Feb '12

06 Feb '12

Forgive me if you receive multiple copies of this and in addition forgive me if sending to this list is seen as crass. I figured you guys would be interested in a workshop directly related to FT and I know most of you personally so figured I could get by with it. I won't spam the contents here, but please goto: http://institute.lanl.gov/resilience/conferences/ftxs2012/ <http://institute.lanl.gov/resilience/conferences/ftxs2012/>We are hosting FTXS in Boston, MA this late June in association with DSN. DSN limits workshop papers to a mere 6 pages so it's a great chance to submit something short. The CFP in text and PDF are at the above link (again, not to spam you email much). Thanks! Cheers! -- Nathan ---------------------------------------------------- Nathan DeBardeleben, Ph.D. Los Alamos National Laboratory High Perf. Computing Systems Integration (HPC-5) Ultra-Scale Research Center, Resilience Lead phone: 505-412-1069 email: ndebard(a)lanl.gov ----------------------------------------------------

1 0

Re: [Mpi3-ft] Fault Tolerance & RMA Discussion
by Josh Hursey 06 Feb '12

06 Feb '12

We are going to meet from 10-11 am (Eastern) on Feb. 7 to continue our conversation. We will use the same call-in information as before. Thanks, Josh On Thu, Feb 2, 2012 at 3:00 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > We made some really good progress on today's call. Attached are some notes > that I took from the call. > > At the end of the call there were a couple of items that we wanted to get > a finer understanding of. As a result we are going to try to setup another > teleconf. > > Below is a doodle poll to pick a date/time: > http://www.doodle.com/kzmiknie8yz4wxkc > > If you are interested in attending this teleconf, please fill out the poll > by 2 pm Eastern on Monday, Feb. 6. > > Thanks, > Josh > > > On Thu, Feb 2, 2012 at 10:01 AM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: > >> Just a reminder that we are meeting today at Noon Eastern to discuss RMA >> in the context of the fault tolerance proposal. >> >> The Run-Through Stabilization proposal can be found attached to the >> ticket: >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… >> >> We will be focusing on section 17.11 of that document. Note that this >> section does not currently explicitly account for the new RMA proposal, but >> we would like to remedy that for the next reading. >> >> Thanks, >> Josh >> >> On Wed, Jan 25, 2012 at 3:15 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: >> >>> There was no one date/time that worked for everyone, but I chose a time >>> that worked for most of the respondents. We will meet Thursday, Feb. 2 from >>> 12-1 pm EST/New York to discuss this topic. >>> >>> We can use the following teleconf information: >>> US Toll Free number: 877-801-8130 >>> Toll number: 1-203-692-8690 >>> Access Code: 1044056 >>> >>> Thanks, >>> Josh >>> >>> >>> On Mon, Jan 23, 2012 at 4:33 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: >>> >>>> (Cross posted to both the RMA and FT MPI-3 listservs) >>>> >>>> During the FT plenary session at the Jan. MPI Forum meeting it was >>>> recommended that some of the members of the FT group and the RMA group have >>>> a meeting to hash out the precise details of the FT semantics for the RMA >>>> chapter. So I would like to facilitate such a discussion, preferability in >>>> the next week (so we have time to fine tune things before the next forum >>>> meeting). >>>> >>>> In general, we are trying to answer the question "How should RMA >>>> operations behave when a process failure occurs?" The feeling seemed to be >>>> that the current approach is ok (invalidating the window, forcing >>>> recreation/validation), but the statement that the memory exposed in the >>>> window is 'undefined' seemed excessive. The suggestion was to change the >>>> wording to something like "Only the memory associated with a window that >>>> was targeted by an operation that modified it is undefined after process >>>> failure in the group associated with the window." This lead to a >>>> considerable amount of debate in the meeting, so it was suggested that we >>>> take the discussion offline. >>>> >>>> Below is a link to a doodle poll to find a good time for a teleconf. If >>>> you are interested in participating in this discussion, please fill this >>>> poll out by 2 PM Eastern on Wed. Jan 25 so we can set the date/time. >>>> http://www.doodle.com/vd33va5h8iankega >>>> >>>> Thanks, >>>> Josh >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> >>> >>> >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] [MPI3-IO] Fault Tolerance & I/O Discussion
by Josh Hursey 06 Feb '12

06 Feb '12

We had a good discussion today. Attached are some notes that I took from the call. There were a few questions that we were discussing at the end of the call. As a result we are going to try to setup another teleconf. Below is a doodle poll to pick a date/time: http://www.doodle.com/zw83eht6hh8mwmic If you are interested in attending this teleconf, please fill out the poll by 2 pm Eastern on Wednesday, Feb. 8. In the mean time let us keep discussing these issues on the FT and I/O mailing lists. My preference would be to just discuss it on the FT mailing list as there are more FT folks over there, and then we would not distract the discussions about the other I/O tickets. Thanks, Josh On Mon, Feb 6, 2012 at 10:55 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Just a reminder that we are meeting today at Noon Eastern. Enclosed are > the call-in details and a link to the FT stabilization proposal. > > Thanks, > Josh > > > On Thu, Feb 2, 2012 at 2:46 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > >> We will meet Monday, Feb. 6 from 12-1 pm EST/New York to discuss I/O in >> the context of the fault tolerance proposal (or the Super Bowl if we get >> bored). >> >> We can use the following teleconf information: >> US Toll Free number: 877-801-8130 >> Toll number: 1-203-692-8690 >> Access Code: 1044056 >> >> The Run-Through Stabilization proposal can be found attached to the >> ticket: >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… >> >> We will be primarily focusing on section 17.12 of that document. I will >> try to send out a reminder the day of. >> >> Thanks, >> Josh >> >> On Tue, Jan 31, 2012 at 5:04 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: >> >>> Try the new link (I had to close the other poll due to a timezone >>> problem): >>> http://www.doodle.com/yifhpi5emyyzrspa >>> >>> -- Josh >>> >>> >>> On Tue, Jan 31, 2012 at 4:41 PM, Mohamad Chaarawi <chaarawi(a)hdfgroup.org >>> > wrote: >>> >>>> Hi Josh, >>>> >>>> When I click the link, it says the poll has been deleted. >>>> >>>> Thanks, >>>> Mohamad >>>> >>>> >>>> On 01/31/2012 11:52 AM, Josh Hursey wrote: >>>> >>>> (Cross posted to both the I/O and FT MPI-3 listservs) >>>> >>>> During the FT plenary session at the Jan. MPI Forum meeting there >>>> were some concerns about fault tolerance semantics in the I/O chapter. We >>>> did not have much time to fully discuss the additional semantics during the >>>> meeting. To make sure that we push towards a complete set of semantics >>>> useful for the I/O community in the next draft I would like to have a >>>> teleconf to discuss the I/O chapter of the FT proposal. Preferability in >>>> the next week and a half (so we have time to fine tune things before the >>>> next forum meeting). >>>> >>>> Below is a link to a doodle poll to find a good time for a teleconf. >>>> If you are interested in participating in this discussion, please fill this >>>> poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. >>>> http://www.doodle.com/s3hz9daeh8pn483m >>>> >>>> Thanks, >>>> Josh >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> >>>> >>>> _______________________________________________ >>>> MPI3-IO mailing [email protected]://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >>>> >>>> >>>> >>>> _______________________________________________ >>>> MPI3-IO mailing list >>>> MPI3-IO(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >>>> >>>> >>> >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] [MPI3-IO] Fault Tolerance & I/O Discussion
by Josh Hursey 06 Feb '12

06 Feb '12

Just a reminder that we are meeting today at Noon Eastern. Enclosed are the call-in details and a link to the FT stabilization proposal. Thanks, Josh On Thu, Feb 2, 2012 at 2:46 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > We will meet Monday, Feb. 6 from 12-1 pm EST/New York to discuss I/O in > the context of the fault tolerance proposal (or the Super Bowl if we get > bored). > > We can use the following teleconf information: > US Toll Free number: 877-801-8130 > Toll number: 1-203-692-8690 > Access Code: 1044056 > > The Run-Through Stabilization proposal can be found attached to the ticket: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 > > https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… > > We will be primarily focusing on section 17.12 of that document. I will > try to send out a reminder the day of. > > Thanks, > Josh > > On Tue, Jan 31, 2012 at 5:04 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: > >> Try the new link (I had to close the other poll due to a timezone >> problem): >> http://www.doodle.com/yifhpi5emyyzrspa >> >> -- Josh >> >> >> On Tue, Jan 31, 2012 at 4:41 PM, Mohamad Chaarawi <chaarawi(a)hdfgroup.org>wrote: >> >>> Hi Josh, >>> >>> When I click the link, it says the poll has been deleted. >>> >>> Thanks, >>> Mohamad >>> >>> >>> On 01/31/2012 11:52 AM, Josh Hursey wrote: >>> >>> (Cross posted to both the I/O and FT MPI-3 listservs) >>> >>> During the FT plenary session at the Jan. MPI Forum meeting there were >>> some concerns about fault tolerance semantics in the I/O chapter. We did >>> not have much time to fully discuss the additional semantics during the >>> meeting. To make sure that we push towards a complete set of semantics >>> useful for the I/O community in the next draft I would like to have a >>> teleconf to discuss the I/O chapter of the FT proposal. Preferability in >>> the next week and a half (so we have time to fine tune things before the >>> next forum meeting). >>> >>> Below is a link to a doodle poll to find a good time for a teleconf. >>> If you are interested in participating in this discussion, please fill this >>> poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. >>> http://www.doodle.com/s3hz9daeh8pn483m >>> >>> Thanks, >>> Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >>> >>> _______________________________________________ >>> MPI3-IO mailing [email protected]://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >>> >>> >>> >>> _______________________________________________ >>> MPI3-IO mailing list >>> MPI3-IO(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >>> >>> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Matching MPI communication object creation across process failure
by Josh Hursey 03 Feb '12

03 Feb '12

Yeah I think that is more of a clarification on (A) - if I understand your email correctly. - If the input communicator is -not- collectively active - Communicator creation operations will fail in error at all alive calling processes - So the creation operations never work around emerging failed processes - If the input communicator is collectively active - Communication creation operations will return successfully, unless - There is an emerging 'new' failure that makes the communicator collectively inactive, in which case the operation will either - Return successfully everywhere if it has passed the decision point in the protocol (at least one alive process returned 'success' already) - Return an error everywhere if it is before the decision point in the protocol. Then the decision is 'return an error' and everyone will decide to return an error. - Once collectively inactive subsequent communicator creation operations will return in error immediately - MPI_Comm_validate is used to reenable the collectives including communicator creation operations. - The MPI_Comm_vallidate when it switches a communicator from collectively inactive to collectively active creates a 'cut' in the message matching. Meaning that collective operations (including communicator creation operations) initiated before the MPI_Comm_validate (under these circumstances) do not need to match across the logical 'cut'. So the user does -not- need to post a matching MPI_Bcast or MPI_Comm_dup after the MPI_Comm_validate if a new failure caused the cut to occur. -- But this gets into more of the issues of (3). How do those semantics sound to folks? -- Josh On Thu, Feb 2, 2012 at 5:28 PM, Sur, Sayantan <sayantan.sur(a)intel.com>wrote: > Hi Josh,**** > > ** ** > > I agree on (1).**** > > ** ** > > Regarding (2), I’m wondering if another case C is feasible. What do you > think?**** > > ** ** > > ( C ) if the input communicator is not required to be collectively active: > MPI_Comm_split returns with error everywhere (all live processes) if it > discovers any existing or emerging failure. i.e. it does not try to ‘work > around’ either type of failure existing or emerging.**** > > ** ** > > i.e. MPI_Comm_split automatically returns “valid” communicators, if the > original communicator had no failures. In case of failure, it simply > returns error. In this case, MPI_Comm_validate() needs to be called in the > error path, not in the error free path.**** > > ** ** > > Thanks.**** > > ** ** > > ===**** > > Sayantan Sur, Ph.D.**** > > Intel Corp.**** > > ** ** > > *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *Josh Hursey > *Sent:* Wednesday, February 01, 2012 2:26 PM > *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > *Subject:* Re: [Mpi3-ft] Matching MPI communication object creation > across process failure**** > > ** ** > > Some notes regarding this thread from the teleconf:**** > > ** ** > > There are three components to the discussion of how communicator creation > operations should behave:**** > > - (1) Uniform creation of the object (created everywhere or no where)**** > > - (2) Requires the input communicator to be collective active**** > > - (3) Matches across emerging process failure**** > > ** ** > > ** ** > > (1) seems to be something we all agree on.**** > > ** ** > > ** ** > > (2) we discussed a bit more on the call. Consider an MPI_Comm_split > operation where the application wants to divide the communicator in half.* > *** > > - (A) If the input communicator is required to be collectively active > then the MPI_Comm_split will return an error if a process fails (makes the > input communicator collective inactive) and the object cannot be uniformly > created. So MPI_Comm_split does not 'work around' emerging failure, but > errors out at the first sign of the failure before the 'decision point' in > the protocol. The 'decision point' being the point in time where the > communication is designated a created.**** > > - (B) if the input communicator is -not- required to be collective active > then the MPI_Comm_split must be able to work around existing and emerging > failure to agree upon the group membership of the output communicator(s). > The resulting communicators may be of unintended size since the > MPI_Comm_split is working around the errors. So the user is forced to call > MPI_Comm_validate after the MPI_Comm_split on the input communicator to > make sure that all processes exited the MPI_Comm_split with acceptable > communicators (if such a distinction is important).**** > > ** ** > > So in (A) the following program is correct:**** > > -------------------------**** > > if( comm_rank < (comm_size/2) ) {**** > > color = 0;**** > > } else {**** > > color = 1;**** > > }**** > > ** ** > > do {**** > > ret = MPI_Comm_split( comm, color, key, &new_comm );**** > > if( ret == MPI_ERR_PROC_FAIL_STOP ) {**** > > /* Re-enable collectives over this communicator */**** > > MPI_Comm_validate( comm, &failed_grp );**** > > MPI_Group_free( &failed_grp );**** > > /* Adjust colors to account for failed processes ***** > > }**** > > /* If there was a process failure, then try again */**** > > } while( ret == MPI_ERR_PROC_FAIL_STOP );**** > > -------------------------**** > > In this example MPI_Comm_split will either return successfully with the > exact groups asked for, or will return an error if a new process failure is > encountered before the communicators are created. One nice thing to point > out is that MPI_Comm_validate() is only called in the error path, and not > in the failure free path.**** > > ** ** > > In (B) it is less likely that MPI_Comm_split will return in error since it > is working around existing and emerging process failure. Since it is > working around emerging failure so it is possible that the resulting > communicators are of a size that is not acceptable to the application. So > the application will need to do some collective, and call MPI_Comm_validate > to ensure that collective completed everywhere before deciding to use the > new communicators.**** > > -------------------------**** > > ...**** > > } while( ret == MPI_ERR_PROC_FAIL_STOP );**** > > /* Collective operation to check if communicators are of acceptable length > */**** > > /* MPI_Comm_validate() to make sure the collective above completed > successfully everywhere */**** > > /* If acceptable then continue */**** > > /* If unacceptable, then destroy them and call MPI_Comm_split again with > new colores */**** > > -------------------------**** > > ** ** > > So (B) pushes the user to call MPI_Comm_validate in the failure free code > path, and do some additional checking. Option (A) keeps the > MPI_Comm_validate in the failure full/error path.**** > > ** ** > > So it seems that requiring the input communicator to be collectively > active (option A) is somewhat better.**** > > ** ** > > ** ** > > Regarding matching across failure (3), this seems to be implied by (1). > Since the uniformity of returning is provided by an agreement protocol. The > question is how difficult would it be to not require matching across > failure (3) if we know that the creation call can is allowed to exit in > error if it encounters an error before agreement (Option 2.A above)? I am > fairly sure that we can implement this safely, but I would like to dwell on > it a bit more to be sure.**** > > ** ** > > ** ** > > What do others think about these points?**** > > ** ** > > -- Josh**** > > ** ** > > On Tue, Jan 31, 2012 at 12:35 PM, Josh Hursey <jjhursey(a)open-mpi.org> > wrote:**** > > Actually the following thread might be more useful for this discussion:*** > * > > http://lists.mpi-forum.org/mpi3-ft/2011/12/0940.php**** > > ** ** > > The example did not come out well in the archives, so below is the diagram > again (hopefully that will work):**** > > So the process stack looks like:**** > > P0 P1**** > > --------------- ----------------**** > > Dup(comm[X-1]) Dup(comm[X-1])**** > > MPI_Allreduce() MPI_Allreduce()**** > > Dup(comm[X]) -> Error**** > > -> Error**** > > ** ** > > So should P1 be required to call Dup(comm[X])?**** > > ** ** > > -- Josh**** > > ** ** > > On Wed, Jan 25, 2012 at 5:08 PM, Josh Hursey <jjhursey(a)open-mpi.org> > wrote:**** > > The current proposal states that MPI object creation functions (e.g., > MPI_Comm_create, MPI_Win_create, MPI_File_open):**** > > -------------------**** > > All participating communicator(s) must be collectively active before > calling any communicator creation operation. Otherwise, the communicator > creation operation will uniformly raise an error code of the class > MPI_ERR_PROC_FAIL_STOP.**** > > ** ** > > If a process failure prevents the uniform creation of the communicator > then the communicator construction operation must ensure that the > communicator is not created, and all alive participating processes will > raise an error code of the class MPI_ERR_PROC_FAIL_STOP. Communicator > construction operations will match across the notification of a process > failure. As such, all alive processes must call the communicator > construction operations the same number of times regardless of whether the > emergent process failure makes the call irrelevant to the application.**** > > -------------------**** > > ** ** > > So there are three points here:**** > > (1) That the communicator must be 'collectively active' before calling > the operation,**** > > (2) Uniform creation of the communication object, and**** > > (3) Creation operations match across process failure.**** > > ** ** > > Point (2) seems to be necessary so that all processes only ever see a > communication object that is consistent across all processes. This implies > a fault tolerant agreement protocol (group membership).**** > > ** ** > > There was a question about why point (3) is necessary. We (Darius, UTK, > and I) discussed this on 12/12/2011, and I posted my notes on this to the > list:**** > > http://lists.mpi-forum.org/mpi3-ft/2011/12/0938.php**** > > Looking back at my written notes, they don't have much more than that to > add to the discussion.**** > > ** ** > > So the problem with (3) seemed to arrise out of the failure handlers, > though I am not convinced that they are strictly to blame in this > circumstance. It seems that the agreement protocol might be factoring into > the discussion as well since it is strongly synchronizing, if not all > processes call the operation how does it know when to bail out. The peer > processes are (a) calling that operation, (b) going to call it but have not > yet, or (c) will never call it because they decided independently not to > base on a locally reported process failure.**** > > ** ** > > It seems that the core problem has to do with when to break out of the > collective creation operation, and when to restore matching.**** > > ** ** > > So should re reconsider the restriction on (3)? More to the point, is it > safe to not require (3)?**** > > ** ** > > -- Josh**** > > ** ** > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey**** > > > > **** > > ** ** > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey**** > > > > **** > > ** ** > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey**** > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Matching MPI communication object creation across process failure
by Sur, Sayantan 02 Feb '12

02 Feb '12

Hi Josh, I agree on (1). Regarding (2), I'm wondering if another case C is feasible. What do you think? ( C ) if the input communicator is not required to be collectively active: MPI_Comm_split returns with error everywhere (all live processes) if it discovers any existing or emerging failure. i.e. it does not try to 'work around' either type of failure existing or emerging. i.e. MPI_Comm_split automatically returns "valid" communicators, if the original communicator had no failures. In case of failure, it simply returns error. In this case, MPI_Comm_validate() needs to be called in the error path, not in the error free path. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Wednesday, February 01, 2012 2:26 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Matching MPI communication object creation across process failure Some notes regarding this thread from the teleconf: There are three components to the discussion of how communicator creation operations should behave: - (1) Uniform creation of the object (created everywhere or no where) - (2) Requires the input communicator to be collective active - (3) Matches across emerging process failure (1) seems to be something we all agree on. (2) we discussed a bit more on the call. Consider an MPI_Comm_split operation where the application wants to divide the communicator in half. - (A) If the input communicator is required to be collectively active then the MPI_Comm_split will return an error if a process fails (makes the input communicator collective inactive) and the object cannot be uniformly created. So MPI_Comm_split does not 'work around' emerging failure, but errors out at the first sign of the failure before the 'decision point' in the protocol. The 'decision point' being the point in time where the communication is designated a created. - (B) if the input communicator is -not- required to be collective active then the MPI_Comm_split must be able to work around existing and emerging failure to agree upon the group membership of the output communicator(s). The resulting communicators may be of unintended size since the MPI_Comm_split is working around the errors. So the user is forced to call MPI_Comm_validate after the MPI_Comm_split on the input communicator to make sure that all processes exited the MPI_Comm_split with acceptable communicators (if such a distinction is important). So in (A) the following program is correct: ------------------------- if( comm_rank < (comm_size/2) ) { color = 0; } else { color = 1; } do { ret = MPI_Comm_split( comm, color, key, &new_comm ); if( ret == MPI_ERR_PROC_FAIL_STOP ) { /* Re-enable collectives over this communicator */ MPI_Comm_validate( comm, &failed_grp ); MPI_Group_free( &failed_grp ); /* Adjust colors to account for failed processes * } /* If there was a process failure, then try again */ } while( ret == MPI_ERR_PROC_FAIL_STOP ); ------------------------- In this example MPI_Comm_split will either return successfully with the exact groups asked for, or will return an error if a new process failure is encountered before the communicators are created. One nice thing to point out is that MPI_Comm_validate() is only called in the error path, and not in the failure free path. In (B) it is less likely that MPI_Comm_split will return in error since it is working around existing and emerging process failure. Since it is working around emerging failure so it is possible that the resulting communicators are of a size that is not acceptable to the application. So the application will need to do some collective, and call MPI_Comm_validate to ensure that collective completed everywhere before deciding to use the new communicators. ------------------------- ... } while( ret == MPI_ERR_PROC_FAIL_STOP ); /* Collective operation to check if communicators are of acceptable length */ /* MPI_Comm_validate() to make sure the collective above completed successfully everywhere */ /* If acceptable then continue */ /* If unacceptable, then destroy them and call MPI_Comm_split again with new colores */ ------------------------- So (B) pushes the user to call MPI_Comm_validate in the failure free code path, and do some additional checking. Option (A) keeps the MPI_Comm_validate in the failure full/error path. So it seems that requiring the input communicator to be collectively active (option A) is somewhat better. Regarding matching across failure (3), this seems to be implied by (1). Since the uniformity of returning is provided by an agreement protocol. The question is how difficult would it be to not require matching across failure (3) if we know that the creation call can is allowed to exit in error if it encounters an error before agreement (Option 2.A above)? I am fairly sure that we can implement this safely, but I would like to dwell on it a bit more to be sure. What do others think about these points? -- Josh On Tue, Jan 31, 2012 at 12:35 PM, Josh Hursey <jjhursey(a)open-mpi.org<mailto:[email protected]>> wrote: Actually the following thread might be more useful for this discussion: http://lists.mpi-forum.org/mpi3-ft/2011/12/0940.php The example did not come out well in the archives, so below is the diagram again (hopefully that will work): So the process stack looks like: P0 P1 --------------- ---------------- Dup(comm[X-1]) Dup(comm[X-1]) MPI_Allreduce() MPI_Allreduce() Dup(comm[X]) -> Error -> Error So should P1 be required to call Dup(comm[X])? -- Josh On Wed, Jan 25, 2012 at 5:08 PM, Josh Hursey <jjhursey(a)open-mpi.org<mailto:[email protected]>> wrote: The current proposal states that MPI object creation functions (e.g., MPI_Comm_create, MPI_Win_create, MPI_File_open): ------------------- All participating communicator(s) must be collectively active before calling any communicator creation operation. Otherwise, the communicator creation operation will uniformly raise an error code of the class MPI_ERR_PROC_FAIL_STOP. If a process failure prevents the uniform creation of the communicator then the communicator construction operation must ensure that the communicator is not created, and all alive participating processes will raise an error code of the class MPI_ERR_PROC_FAIL_STOP. Communicator construction operations will match across the notification of a process failure. As such, all alive processes must call the communicator construction operations the same number of times regardless of whether the emergent process failure makes the call irrelevant to the application. ------------------- So there are three points here: (1) That the communicator must be 'collectively active' before calling the operation, (2) Uniform creation of the communication object, and (3) Creation operations match across process failure. Point (2) seems to be necessary so that all processes only ever see a communication object that is consistent across all processes. This implies a fault tolerant agreement protocol (group membership). There was a question about why point (3) is necessary. We (Darius, UTK, and I) discussed this on 12/12/2011, and I posted my notes on this to the list: http://lists.mpi-forum.org/mpi3-ft/2011/12/0938.php Looking back at my written notes, they don't have much more than that to add to the discussion. So the problem with (3) seemed to arrise out of the failure handlers, though I am not convinced that they are strictly to blame in this circumstance. It seems that the agreement protocol might be factoring into the discussion as well since it is strongly synchronizing, if not all processes call the operation how does it know when to bail out. The peer processes are (a) calling that operation, (b) going to call it but have not yet, or (c) will never call it because they decided independently not to base on a locally reported process failure. It seems that the core problem has to do with when to break out of the collective creation operation, and when to restore matching. So should re reconsider the restriction on (3)? More to the point, is it safe to not require (3)? -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Fault Tolerance & RMA Discussion
by Josh Hursey 02 Feb '12

02 Feb '12

We made some really good progress on today's call. Attached are some notes that I took from the call. At the end of the call there were a couple of items that we wanted to get a finer understanding of. As a result we are going to try to setup another teleconf. Below is a doodle poll to pick a date/time: http://www.doodle.com/kzmiknie8yz4wxkc If you are interested in attending this teleconf, please fill out the poll by 2 pm Eastern on Monday, Feb. 6. Thanks, Josh On Thu, Feb 2, 2012 at 10:01 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Just a reminder that we are meeting today at Noon Eastern to discuss RMA > in the context of the fault tolerance proposal. > > The Run-Through Stabilization proposal can be found attached to the ticket: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 > > https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… > > We will be focusing on section 17.11 of that document. Note that this > section does not currently explicitly account for the new RMA proposal, but > we would like to remedy that for the next reading. > > Thanks, > Josh > > On Wed, Jan 25, 2012 at 3:15 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: > >> There was no one date/time that worked for everyone, but I chose a time >> that worked for most of the respondents. We will meet Thursday, Feb. 2 from >> 12-1 pm EST/New York to discuss this topic. >> >> We can use the following teleconf information: >> US Toll Free number: 877-801-8130 >> Toll number: 1-203-692-8690 >> Access Code: 1044056 >> >> Thanks, >> Josh >> >> >> On Mon, Jan 23, 2012 at 4:33 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: >> >>> (Cross posted to both the RMA and FT MPI-3 listservs) >>> >>> During the FT plenary session at the Jan. MPI Forum meeting it was >>> recommended that some of the members of the FT group and the RMA group have >>> a meeting to hash out the precise details of the FT semantics for the RMA >>> chapter. So I would like to facilitate such a discussion, preferability in >>> the next week (so we have time to fine tune things before the next forum >>> meeting). >>> >>> In general, we are trying to answer the question "How should RMA >>> operations behave when a process failure occurs?" The feeling seemed to be >>> that the current approach is ok (invalidating the window, forcing >>> recreation/validation), but the statement that the memory exposed in the >>> window is 'undefined' seemed excessive. The suggestion was to change the >>> wording to something like "Only the memory associated with a window that >>> was targeted by an operation that modified it is undefined after process >>> failure in the group associated with the window." This lead to a >>> considerable amount of debate in the meeting, so it was suggested that we >>> take the discussion offline. >>> >>> Below is a link to a doodle poll to find a good time for a teleconf. If >>> you are interested in participating in this discussion, please fill this >>> poll out by 2 PM Eastern on Wed. Jan 25 so we can set the date/time. >>> http://www.doodle.com/vd33va5h8iankega >>> >>> Thanks, >>> Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Meeting(s) next week
by Josh Hursey 02 Feb '12

02 Feb '12

How about we plan on meeting Tuesday, Feb. 7 at Noon Eastern to talk about primarily about the RTS proposal. Then meet again on Friday, Feb. 10 at Noon Eastern and we will split the time between the RTS proposal, Tony's proposal, and Piggybacking. I'll send out reminders, but we will use the same teleconf information (I'll let you know if for some reason we have to change the number). Note that on Monday, Feb. 6 there is a Noon Eastern teleconf with the I/O working group for those interested in that discussion. I know this is starting to become a bit time intensive, and I appreciate the time that each of you are able to give to the effort. I think we are making some good, solid progress on the proposals at hand and should have a more productive session at the MPI Forum in March as a result. Thanks, Josh On Wed, Feb 1, 2012 at 5:19 PM, Darius Buntinas <buntinas(a)mcs.anl.gov>wrote: > Feb 8 and 10 both work for me. > > -d > > On Feb 1, 2012, at 3:52 PM, Josh Hursey wrote: > > > I have a traveling conflict next Wednesday, so I will not be able to > attend if we have our regular teleconf on Feb. 8. On the call today, I > suggested moving the teleconf to Tuesday, Feb. 7 at the same time (Noon > Eastern). Would this be ok with folks? > > > > Tony: I know that you wanted to talk about your proposal during the > meeting next week, would this work for your or would keeping the meeting on > Feb. 8 be preferable? > > > > On a second note, it seems that we still have a number of things to > discuss. To make some more progress, would folks be available to meet on > Friday, Feb. 10 at the usual time (Noon Eastern)? > > > > Let me know, and I'll set things up for next week. > > > > Thanks, > > Josh > > > > -- > > Joshua Hursey > > Postdoctoral Research Associate > > Oak Ridge National Laboratory > > http://users.nccs.gov/~jjhursey > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0