February 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] [MPI3-IO] Fault Tolerance & I/O Discussion
by Josh Hursey 02 Feb '12

02 Feb '12

We will meet Monday, Feb. 6 from 12-1 pm EST/New York to discuss I/O in the context of the fault tolerance proposal (or the Super Bowl if we get bored). We can use the following teleconf information: US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 The Run-Through Stabilization proposal can be found attached to the ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… We will be primarily focusing on section 17.12 of that document. I will try to send out a reminder the day of. Thanks, Josh On Tue, Jan 31, 2012 at 5:04 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Try the new link (I had to close the other poll due to a timezone problem): > http://www.doodle.com/yifhpi5emyyzrspa > > -- Josh > > > On Tue, Jan 31, 2012 at 4:41 PM, Mohamad Chaarawi <chaarawi(a)hdfgroup.org>wrote: > >> Hi Josh, >> >> When I click the link, it says the poll has been deleted. >> >> Thanks, >> Mohamad >> >> >> On 01/31/2012 11:52 AM, Josh Hursey wrote: >> >> (Cross posted to both the I/O and FT MPI-3 listservs) >> >> During the FT plenary session at the Jan. MPI Forum meeting there were >> some concerns about fault tolerance semantics in the I/O chapter. We did >> not have much time to fully discuss the additional semantics during the >> meeting. To make sure that we push towards a complete set of semantics >> useful for the I/O community in the next draft I would like to have a >> teleconf to discuss the I/O chapter of the FT proposal. Preferability in >> the next week and a half (so we have time to fine tune things before the >> next forum meeting). >> >> Below is a link to a doodle poll to find a good time for a teleconf. If >> you are interested in participating in this discussion, please fill this >> poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. >> http://www.doodle.com/s3hz9daeh8pn483m >> >> Thanks, >> Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> MPI3-IO mailing [email protected]://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> >> >> _______________________________________________ >> MPI3-IO mailing list >> MPI3-IO(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Fault Tolerance & RMA Discussion
by Josh Hursey 02 Feb '12

02 Feb '12

Just a reminder that we are meeting today at Noon Eastern to discuss RMA in the context of the fault tolerance proposal. The Run-Through Stabilization proposal can be found attached to the ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… We will be focusing on section 17.11 of that document. Note that this section does not currently explicitly account for the new RMA proposal, but we would like to remedy that for the next reading. Thanks, Josh On Wed, Jan 25, 2012 at 3:15 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > There was no one date/time that worked for everyone, but I chose a time > that worked for most of the respondents. We will meet Thursday, Feb. 2 from > 12-1 pm EST/New York to discuss this topic. > > We can use the following teleconf information: > US Toll Free number: 877-801-8130 > Toll number: 1-203-692-8690 > Access Code: 1044056 > > Thanks, > Josh > > > On Mon, Jan 23, 2012 at 4:33 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: > >> (Cross posted to both the RMA and FT MPI-3 listservs) >> >> During the FT plenary session at the Jan. MPI Forum meeting it was >> recommended that some of the members of the FT group and the RMA group have >> a meeting to hash out the precise details of the FT semantics for the RMA >> chapter. So I would like to facilitate such a discussion, preferability in >> the next week (so we have time to fine tune things before the next forum >> meeting). >> >> In general, we are trying to answer the question "How should RMA >> operations behave when a process failure occurs?" The feeling seemed to be >> that the current approach is ok (invalidating the window, forcing >> recreation/validation), but the statement that the memory exposed in the >> window is 'undefined' seemed excessive. The suggestion was to change the >> wording to something like "Only the memory associated with a window that >> was targeted by an operation that modified it is undefined after process >> failure in the group associated with the window." This lead to a >> considerable amount of debate in the meeting, so it was suggested that we >> take the discussion offline. >> >> Below is a link to a doodle poll to find a good time for a teleconf. If >> you are interested in participating in this discussion, please fill this >> poll out by 2 PM Eastern on Wed. Jan 25 so we can set the date/time. >> http://www.doodle.com/vd33va5h8iankega >> >> Thanks, >> Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Matching MPI communication object creation across process failure
by Josh Hursey 01 Feb '12

01 Feb '12

Some notes regarding this thread from the teleconf: There are three components to the discussion of how communicator creation operations should behave: - (1) Uniform creation of the object (created everywhere or no where) - (2) Requires the input communicator to be collective active - (3) Matches across emerging process failure (1) seems to be something we all agree on. (2) we discussed a bit more on the call. Consider an MPI_Comm_split operation where the application wants to divide the communicator in half. - (A) If the input communicator is required to be collectively active then the MPI_Comm_split will return an error if a process fails (makes the input communicator collective inactive) and the object cannot be uniformly created. So MPI_Comm_split does not 'work around' emerging failure, but errors out at the first sign of the failure before the 'decision point' in the protocol. The 'decision point' being the point in time where the communication is designated a created. - (B) if the input communicator is -not- required to be collective active then the MPI_Comm_split must be able to work around existing and emerging failure to agree upon the group membership of the output communicator(s). The resulting communicators may be of unintended size since the MPI_Comm_split is working around the errors. So the user is forced to call MPI_Comm_validate after the MPI_Comm_split on the input communicator to make sure that all processes exited the MPI_Comm_split with acceptable communicators (if such a distinction is important). So in (A) the following program is correct: ------------------------- if( comm_rank < (comm_size/2) ) { color = 0; } else { color = 1; } do { ret = MPI_Comm_split( comm, color, key, &new_comm ); if( ret == MPI_ERR_PROC_FAIL_STOP ) { /* Re-enable collectives over this communicator */ MPI_Comm_validate( comm, &failed_grp ); MPI_Group_free( &failed_grp ); /* Adjust colors to account for failed processes * } /* If there was a process failure, then try again */ } while( ret == MPI_ERR_PROC_FAIL_STOP ); ------------------------- In this example MPI_Comm_split will either return successfully with the exact groups asked for, or will return an error if a new process failure is encountered before the communicators are created. One nice thing to point out is that MPI_Comm_validate() is only called in the error path, and not in the failure free path. In (B) it is less likely that MPI_Comm_split will return in error since it is working around existing and emerging process failure. Since it is working around emerging failure so it is possible that the resulting communicators are of a size that is not acceptable to the application. So the application will need to do some collective, and call MPI_Comm_validate to ensure that collective completed everywhere before deciding to use the new communicators. ------------------------- ... } while( ret == MPI_ERR_PROC_FAIL_STOP ); /* Collective operation to check if communicators are of acceptable length */ /* MPI_Comm_validate() to make sure the collective above completed successfully everywhere */ /* If acceptable then continue */ /* If unacceptable, then destroy them and call MPI_Comm_split again with new colores */ ------------------------- So (B) pushes the user to call MPI_Comm_validate in the failure free code path, and do some additional checking. Option (A) keeps the MPI_Comm_validate in the failure full/error path. So it seems that requiring the input communicator to be collectively active (option A) is somewhat better. Regarding matching across failure (3), this seems to be implied by (1). Since the uniformity of returning is provided by an agreement protocol. The question is how difficult would it be to not require matching across failure (3) if we know that the creation call can is allowed to exit in error if it encounters an error before agreement (Option 2.A above)? I am fairly sure that we can implement this safely, but I would like to dwell on it a bit more to be sure. What do others think about these points? -- Josh On Tue, Jan 31, 2012 at 12:35 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Actually the following thread might be more useful for this discussion: > http://lists.mpi-forum.org/mpi3-ft/2011/12/0940.php > > The example did not come out well in the archives, so below is the diagram > again (hopefully that will work): > So the process stack looks like: > P0 P1 > --------------- ---------------- > Dup(comm[X-1]) Dup(comm[X-1]) > MPI_Allreduce() MPI_Allreduce() > Dup(comm[X]) -> Error > -> Error > > So should P1 be required to call Dup(comm[X])? > > -- Josh > > On Wed, Jan 25, 2012 at 5:08 PM, Josh Hursey <jjhursey(a)open-mpi.org>wrote: > >> The current proposal states that MPI object creation functions (e.g., >> MPI_Comm_create, MPI_Win_create, MPI_File_open): >> ------------------- >> All participating communicator(s) must be collectively active before >> calling any communicator creation operation. Otherwise, the communicator >> creation operation will uniformly raise an error code of the class >> MPI_ERR_PROC_FAIL_STOP. >> >> If a process failure prevents the uniform creation of the communicator >> then the communicator construction operation must ensure that the >> communicator is not created, and all alive participating processes will >> raise an error code of the class MPI_ERR_PROC_FAIL_STOP. Communicator >> construction operations will match across the notification of a process >> failure. As such, all alive processes must call the communicator >> construction operations the same number of times regardless of whether the >> emergent process failure makes the call irrelevant to the application. >> ------------------- >> >> So there are three points here: >> (1) That the communicator must be 'collectively active' before calling >> the operation, >> (2) Uniform creation of the communication object, and >> (3) Creation operations match across process failure. >> >> Point (2) seems to be necessary so that all processes only ever see a >> communication object that is consistent across all processes. This implies >> a fault tolerant agreement protocol (group membership). >> >> There was a question about why point (3) is necessary. We (Darius, UTK, >> and I) discussed this on 12/12/2011, and I posted my notes on this to the >> list: >> http://lists.mpi-forum.org/mpi3-ft/2011/12/0938.php >> Looking back at my written notes, they don't have much more than that to >> add to the discussion. >> >> So the problem with (3) seemed to arrise out of the failure handlers, >> though I am not convinced that they are strictly to blame in this >> circumstance. It seems that the agreement protocol might be factoring into >> the discussion as well since it is strongly synchronizing, if not all >> processes call the operation how does it know when to bail out. The peer >> processes are (a) calling that operation, (b) going to call it but have not >> yet, or (c) will never call it because they decided independently not to >> base on a locally reported process failure. >> >> It seems that the core problem has to do with when to break out of the >> collective creation operation, and when to restore matching. >> >> So should re reconsider the restriction on (3)? More to the point, is it >> safe to not require (3)? >> >> -- Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Meeting(s) next week
by Darius Buntinas 01 Feb '12

01 Feb '12

Feb 8 and 10 both work for me. -d On Feb 1, 2012, at 3:52 PM, Josh Hursey wrote: > I have a traveling conflict next Wednesday, so I will not be able to attend if we have our regular teleconf on Feb. 8. On the call today, I suggested moving the teleconf to Tuesday, Feb. 7 at the same time (Noon Eastern). Would this be ok with folks? > > Tony: I know that you wanted to talk about your proposal during the meeting next week, would this work for your or would keeping the meeting on Feb. 8 be preferable? > > On a second note, it seems that we still have a number of things to discuss. To make some more progress, would folks be available to meet on Friday, Feb. 10 at the usual time (Noon Eastern)? > > Let me know, and I'll set things up for next week. > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Meeting(s) next week
by Anthony Skjellum 01 Feb '12

01 Feb '12

Feb 10 is when we would like to talk. Feb 7 we are booked! Anthony Skjellum, PhD 205-807-4968 On Feb 1, 2012, at 4:59 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > I have a traveling conflict next Wednesday, so I will not be able to attend if we have our regular teleconf on Feb. 8. On the call today, I suggested moving the teleconf to Tuesday, Feb. 7 at the same time (Noon Eastern). Would this be ok with folks? > > Tony: I know that you wanted to talk about your proposal during the meeting next week, would this work for your or would keeping the meeting on Feb. 8 be preferable? > > On a second note, it seems that we still have a number of things to discuss. To make some more progress, would folks be available to meet on Friday, Feb. 10 at the usual time (Noon Eastern)? > > Let me know, and I'll set things up for next week. > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Meeting(s) next week
by Josh Hursey 01 Feb '12

01 Feb '12

I have a traveling conflict next Wednesday, so I will not be able to attend if we have our regular teleconf on Feb. 8. On the call today, I suggested moving the teleconf to Tuesday, Feb. 7 at the same time (Noon Eastern). Would this be ok with folks? Tony: I know that you wanted to talk about your proposal during the meeting next week, would this work for your or would keeping the meeting on Feb. 8 be preferable? On a second note, it seems that we still have a number of things to discuss. To make some more progress, would folks be available to meet on Friday, Feb. 10 at the usual time (Noon Eastern)? Let me know, and I'll set things up for next week. Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0