July 2011 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] NOAA GPU work
by George R Carr Jr 29 Jul '11

29 Jul '11

During the meetings in Chicago, there were a couple questions regarding NOAA's use of GPUs. The GPU work is being performed by people in the same group that I am part of. While we were in Chicago, Tom Henderson was at the 2011 Symposium on Application Accelerators in High-Performance Computing (SAAHPC'11) presenting some of our more recent results. Although the primary thrust of the paper/presentation focused on compilers, the results clearly justify further work with this architecture for the weather community. See http://saahpc.ncsa.illinois.edu for additional information. I have attached Tom's paper and presentation. These are available publically on the conference web so you can feel free to do what you may with them. Regards, George P.S. Alternative contact information georgercarrjr(a)acm.org 303-325-3334 -- George R Carr Jr U.S. Department of Commerce NOAA/OAR/GSD, DSRC 2B142 Tel: 303-497-4714 Email: George.Carr(a)noaa.gov

1 0

Re: [Mpi3-ft] Working group wiki pages broken?
by Josh Hursey 28 Jul '11

28 Jul '11

Things should be back up now. Let me know if you see any other issues and I can forward them to the sysadmin. -- Josh On Thu, Jul 28, 2011 at 8:59 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Indiana University is apparently upgrading their subversion/trac > server this morning. The admin is scheduled to bring things back > online later this morning. > > On Thu, Jul 28, 2011 at 8:54 AM, Brian Smith <smithbr(a)us.ibm.com> wrote: >> Hi all, I'm getting: >> Trac Error >> >> TracError: The Trac Environment needs to be upgraded. >> >> Run "trac-admin /opt/trac-osl/mpi-forum-web upgrade" >> >> when I click on the FT working group wiki pages link from >> http://meetings.mpi-forum.org/mpi3.0_ft.php >> >> Is anyone else seeing this too? >> >> Thanks. >> >> >> Brian Smith (smithbr(a)us.ibm.com) >> BlueGene MPI Development/ >> Communications Team Lead >> IBM Master Inventor >> IBM Rochester >> Phone: 507 253 4717 >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Working group wiki pages broken?
by Josh Hursey 28 Jul '11

28 Jul '11

Indiana University is apparently upgrading their subversion/trac server this morning. The admin is scheduled to bring things back online later this morning. On Thu, Jul 28, 2011 at 8:54 AM, Brian Smith <smithbr(a)us.ibm.com> wrote: > Hi all, I'm getting: > Trac Error > > TracError: The Trac Environment needs to be upgraded. > > Run "trac-admin /opt/trac-osl/mpi-forum-web upgrade" > > when I click on the FT working group wiki pages link from > http://meetings.mpi-forum.org/mpi3.0_ft.php > > Is anyone else seeing this too? > > Thanks. > > > Brian Smith (smithbr(a)us.ibm.com) > BlueGene MPI Development/ > Communications Team Lead > IBM Master Inventor > IBM Rochester > Phone: 507 253 4717 > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Working group wiki pages broken?
by Brian Smith 28 Jul '11

28 Jul '11

Hi all, I'm getting: Trac Error TracError: The Trac Environment needs to be upgraded. Run "trac-admin /opt/trac-osl/mpi-forum-web upgrade" when I click on the FT working group wiki pages link from http://meetings.mpi-forum.org/mpi3.0_ft.php Is anyone else seeing this too? Thanks. Brian Smith (smithbr(a)us.ibm.com) BlueGene MPI Development/ Communications Team Lead IBM Master Inventor IBM Rochester Phone: 507 253 4717

1 0

Re: [Mpi3-ft] Notes from RTS Plenary session at July 2011 MPI Forum
by Joshua Hursey 23 Jul '11

23 Jul '11

I'll just plan to meet next on 8/3. In the between time try to read through the notes, and think about some alternative designs that might address some of the high level concerns. Thanks, Josh On Jul 22, 2011, at 3:13 PM, Howard Pritchard wrote: > Hi Josh, > > I could make it next week. > > Howard > > Joshua Hursey wrote: >> Attached are the notes from the Run-Through Stabilization Plenary >> sessions at the MPI Forum Meeting this week in Chicago. I would like to >> thank everyone that helped take notes in the meeting. The attached >> represents the sum of all of those notes sent both to the list and >> directly to me. >> >> We received some good feedback from the MPI Forum. They gave us a bunch >> of things to think about and work through. We anticipated that we would >> receive a bunch of feedback both small and large. Ultimately this will >> make the proposal stronger, and better suited for the applications. >> >> Our next scheduled teleconf is 8/3/2011 (2 weeks). If folks would like >> to meet next week we can. I would like to spend the next teleconf >> regrouping and talking through some of the suggestions made by the MPI >> Forum. >> >> Thanks, >> Josh >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > -- > Howard Pritchard > Software Engineering > Cray, Inc. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Notes from RTS Plenary session at July 2011 MPI Forum
by Howard Pritchard 22 Jul '11

22 Jul '11

Hi Josh, I could make it next week. Howard Joshua Hursey wrote: > Attached are the notes from the Run-Through Stabilization Plenary > sessions at the MPI Forum Meeting this week in Chicago. I would like to > thank everyone that helped take notes in the meeting. The attached > represents the sum of all of those notes sent both to the list and > directly to me. > > We received some good feedback from the MPI Forum. They gave us a bunch > of things to think about and work through. We anticipated that we would > receive a bunch of feedback both small and large. Ultimately this will > make the proposal stronger, and better suited for the applications. > > Our next scheduled teleconf is 8/3/2011 (2 weeks). If folks would like > to meet next week we can. I would like to spend the next teleconf > regrouping and talking through some of the suggestions made by the MPI > Forum. > > Thanks, > Josh > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-ft] Notes from RTS Plenary session at July 2011 MPI Forum
by Darius Buntinas 22 Jul '11

22 Jul '11

I won't be available next week, but feel free to meet without me. -d On Jul 22, 2011, at 11:23 AM, Joshua Hursey wrote: > Attached are the notes from the Run-Through Stabilization Plenary sessions at the MPI Forum Meeting this week in Chicago. I would like to thank everyone that helped take notes in the meeting. The attached represents the sum of all of those notes sent both to the list and directly to me. > > We received some good feedback from the MPI Forum. They gave us a bunch of things to think about and work through. We anticipated that we would receive a bunch of feedback both small and large. Ultimately this will make the proposal stronger, and better suited for the applications. > > Our next scheduled teleconf is 8/3/2011 (2 weeks). If folks would like to meet next week we can. I would like to spend the next teleconf regrouping and talking through some of the suggestions made by the MPI Forum. > > Thanks, > Josh > > <07-2011-RTS-MPI-Forum-Notes.txt>_______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Notes from RTS Plenary session at July 2011 MPI Forum
by Joshua Hursey 22 Jul '11

22 Jul '11

Attached are the notes from the Run-Through Stabilization Plenary sessions at the MPI Forum Meeting this week in Chicago. I would like to thank everyone that helped take notes in the meeting. The attached represents the sum of all of those notes sent both to the list and directly to me. We received some good feedback from the MPI Forum. They gave us a bunch of things to think about and work through. We anticipated that we would receive a bunch of feedback both small and large. Ultimately this will make the proposal stronger, and better suited for the applications. Our next scheduled teleconf is 8/3/2011 (2 weeks). If folks would like to meet next week we can. I would like to spend the next teleconf regrouping and talking through some of the suggestions made by the MPI Forum. Thanks, Josh

1 0

Re: [Mpi3-ft] radical idea?
by Graham, Richard L. 21 Jul '11

21 Jul '11

Comments in line. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas Sent: Thursday, July 21, 2011 2:43 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] radical idea? Another issue someone brought up was that communicators were immutable and our proposal would change that by adding process state info. Of course communicators aren't exactly immutable currently (e.g., we can set error handlers), but I guess the process state info that we're proposing is expected to be much more dynamic than error handlers (so maybe we were making the communicators "too mutable"??). [rich] we are getting into a world of computing that is more and more mutable - the questions is how to respond to such changes (process failure in this case) in a way that makes it convenient for the app to continue, and minimizes lost work. There is NOT going to be a single way to do this for all apps - we need to provide a reasonable range of options. I'll expand a little on what Dave said about thread safety. The idea was that calling MPI_Comm_validate[_all] changed some implicit state associated with the communicator, i.e., the "snapshot" of failed processes, and so one thread could call get while another was calling validate, and the "new" flag of validate is only meaningful if you know the last time it was called, which you may not (easily) if it's multithreaded. An alternative that was suggested, and that my previous email showed, was to return a handle to the "snapshot", so you could have multiple such "snapshots", and threads would be able to query them, etc., individually. This removes the mutability concerns. [rich] I believe this is an issue with atomicity, not different than any other issue with atomicity. If one does things in a way that is not thread safe, you get what you ask for. A library can protect such accesses, but can't prevent (actually does not want to prevent) users from making calls that are not thread safe. However, we can't use these individual "snapshots" for setting processes to NULL, otherwise we'd have to pass the communicator, rank and "snapshot" to, e.g., MPI_Send in order for the library to know whether to treat the process as NULL. So, the "snapshot" only keeps process UP/DOWN state. To address the PROC_NULL semantics, I suggested the MPI_COMM_NULLIFY function that set PROC_NULL semantics for a process in a communicator. This can be used independently of a "snapshot" (and the rest of the ft proposal, if it makes any sense to do so), so you can technically set a live process to NULL. This functionality may be (is) more controversial, so I figure we can put this into a separate proposal. Also, I think it may be easier to explain the concepts separately. We were running out of time towards the end, so we weren't able to spend as much time on getting feedback on collectives, but there was some resistance to having ft collectives and whether it was useful to have a reduce with an empty value (vs. creating a new communicator and doing collectives on that). So I don't know whether people just didn't get it and need to think about it more or whether people are fundamentally opposed to ft collectives. [rich] I will come back to what I said before, for some cases the ft collectives make sense, and for others it does not. It really depends on what the app is trying to do. This is one reason we have been advocating all along two different sets of collectives - one that guarantees all will get the "same" answer (if all are supposed to), and a set that does not, but lets the user use the validate call to check for the state of the procs. This was actually a request for the apps in the very early days of the FT discussions, I guess 3+ years ago, when we met at the Marriott O'Hare... Rich -d On Jul 21, 2011, at 12:57 PM, Solt, David George wrote: > I think that the main objection was not to the MPI_PROC_NULL semantics themselves, but the fact that querying and setting the state of a communicator is not thread safe. One thread can ask how many ranks are in a given state on comm A and another rank can then change the state of the comm by setting a rank to MPI_PROC_NULL, invalidating the results of the first thread's query. If we remove the ability to explicitly change the state of a rank, then notification can be handled in a more thread-safe way. > > Our argument was that separate threads should not be working with the same comm at the same time as this doesn't work well currently (e.g. two threads trying to call collectives on the same comm at the same time). > > In general, I think that if there was a way to do what we wanted without introducing 40? state quering/setting functions we would get less pushback. Getting rid of the MPI_PROC_NULL state opens up several options to such a simplification. > > [[ personal opinion: I think MPI_PROC_NULL semantics is a nice touch, though it is a guess how much it would actually get used vs. immediately calling MPI_Comm_split or someday calling MPI_Comm_restore. I'd be willing to drop it if it allowed us to move forward. ]] > > Thanks, > Dave > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. > Sent: Thursday, July 21, 2011 12:46 PM > To: 'MPI 3.0 Fault Tolerance and Dynamic Process Control working Group' > Subject: Re: [Mpi3-ft] radical idea? > > Since I missed the meeting - what were the objections people had to the proc null semantics ? Is the suggestion that the app explicitly deal with the missing rank all the time, or something else ? What was the motivation ? What were the problems they saw ? What were the alternative suggestions ? > > Rich > > > > Sent with Good (www . good . com) > > > -----Original Message----- > From: Howard Pritchard [mailto:[email protected]] > Sent: Thursday, July 21, 2011 01:08 PM Eastern Standard Time > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] radical idea? > > Hi Darius, > > If we want to get something about RTS into MPI 3.0 I don't > think we have time to manage it as a set of smaller proposals. > > If we can eliminate the state problem that bothered some at > the last forum meeting that would be a good start. Also, > if we could simplify the proposal some by removing > the PROC_NULL semantics, I would be in favor of that. > > If we want to limit the use of RTS to a small number of use > cases (like the NOAA example), then I could see deferring > "repairable" communicators to 3.1. > > Howard > > Darius Buntinas wrote: >> We could break the rts proposal into smaller ones: >> >> point-to-point: local up/down checks; errors on sending to failed processes >> recognition/PROC_NULLification: Add a function to set a rank in a communicator to >> MPI_PROC_NULL >> fault-aware collectives: collectives don't hang, but they're permanently broken once a >> proc in the communicator fails >> "repairable" collectives: validate_all; collectives can be reactivated after failure >> >> I don't think anyone really objected to "point-to-point" or "fault-aware collectives". We'll have to work on the others. >> >> -d >> >> >> On Jul 20, 2011, at 9:13 AM, Joshua Hursey wrote: >> >>> I'll have to think a bit more and come back to this thread. But I wanted to interject something I was thinking about on the plane ride back. What if we removed the notion of recognized failures? >>> >>> This was a point that was mentioned a couple times in discussion - that we have a bunch of functions and extra state on each communicator because we want to allow the application to recognize failures to get PROC_NULL semantics. If we remove the notion of recognized failures, then the up/down state on the group would be enough to track. So failed processes will always return an error regardless of if the failure has been 'seen' by the application before or not. >>> >>> The state of a process would be able to change as MPI finds out about new failures. But we can provide a 'state snapshot' object (which was mentioned in discussion, and I think is what Darius is getting at below) to allow for more consistent lookups if the application so desires. This removes the local/global list tracking on each handle, and moves it to a separate object that the user is in control of. The user can still reference the best known state if they are not concerned about consistency (e.g., MPI_Comm_validate_get_state(comm, ...) vs MPI_Snapshot_validate_get_state(snapshot_handle, ...)). >>> >>> Some applications would like the PROC_NULL semantics. But if we can convince ourselves that a library on top of MPI could provide those (by adding the proc_null check in the PMPI layer), then we might be able to reduce the complexity of the proposal by pushing some of the state tracking responsibility above MPI. >>> >>> I still have not figured out the implications on an application using collective operations if we remove the NULL state, but it is something to think about. >>> >>> -- Josh >>> >>> On Jul 19, 2011, at 6:11 PM, Darius Buntinas wrote: >>> >>>> The MPI_COMM_NULLIFY() function would effectively set the process to MPI_PROC_STATE_NULL. >>>> >>>> In the proposal we had MPI_PROC_STATE_NULL, _FAILED and _OK. I'm proposing separating NULL from FAILED and OK. So the MPI_COMM_GET_STATE() function (and friends) would let you query the (locally known) FAILED/OK state of the process, while MPI_COMM_NULLIFY() (and friends) would let you set the process to NULL. There would be essentially two state variables associated with each process: one indicating whether it's failed or not (let's call it LIVENESS), and the other whether it's has PROC_NULL semantics (call it NULLIFICATION). The LIVENESS state is controlled by the MPI library while the NULLIFICATION state is controlled by the user. The table below shows how these states would match up with the current proposal: >>>> >>>> Current proposal state LIVENESS NULLIFICATION >>>> -----------------------+----------+--------------- >>>> MPI_PROC_STATE_OK OK NORMAL >>>> MPI_PROC_STATE_FAILED FAILED NORMAL >>>> MPI_PROC_STATE_NULL FAILED NULL >>>> <UNDEFINED> OK NULL >>>> >>>> Notice that there's a combination possible that's not covered by the current proposal. I'm not sure whether that's a useful state (or if we should disallow it). >>>> >>>> We'd could add a function to set the NULLIFICATION state from NULL to NORMAL for completeness. >>>> >>>> -d >>>> >>>> >>>> On Jul 19, 2011, at 4:32 PM, Solt, David George wrote: >>>> >>>>> This works for "reading" state, but has no way to set a processes state. (Not sure how radical your trying to go here... is part of the change here that there would no longer be a MPI_PROC_STATE_NULL state?) >>>>> Dave >>>>> >>>>> -----Original Message----- >>>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>>> Sent: Tuesday, July 19, 2011 3:17 PM >>>>> To: Darius Buntinas >>>>> Cc: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>>> Subject: Re: [Mpi3-ft] radical idea? >>>>> >>>>> >>>>> Howard pointed out that I forgot to add a FREE operation: >>>>> >>>>> MPI_PROC_STATE_FREE(state_handle) >>>>> INOUT: MPI_PROC_STATE state_handle >>>>> >>>>> -d >>>>> >>>>> On Jul 19, 2011, at 3:07 PM, Darius Buntinas wrote: >>>>> >>>>>> MPI_COMM_GET_STATE(comm, state_handle) >>>>>> IN: MPI_COMM comm >>>>>> OUT: MPI_PROC_STATE state_handle >>>>>> and ditto for GROUP, FILE, WIN as necessary >>>>>> >>>>>> MPI_GET_PROC_STATE_SIZE(state_handle, mask, size) >>>>>> IN: MPI_PROC_STATE state_handle >>>>>> IN: int mask >>>>>> OUT: int size >>>>>> >>>>>> MPI_GET_PROC_STATE_LIST(state_handle, mask, list) >>>>>> IN: MPI_PROC_STATE state_handle >>>>>> IN: int mask >>>>>> OUT: int list[] >>>>>> >>>>>> MPI_GET_PROC_STATE_NEW(state_handle1, state_handle2, state_handle_new) >>>>>> IN: MPI_PROC_STATE state_handle1 >>>>>> IN: MPI_PROC_STATE state_handle2 >>>>>> OUT: MPI_PROC_STATE state_handle_new >>>>>> This gives newly failed processes in state_handle2 since state_handle1. >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > -- > Howard Pritchard > Software Engineering > Cray, Inc. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] radical idea?
by Graham, Richard L. 21 Jul '11

21 Jul '11

Comments in line -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Solt, David George Sent: Thursday, July 21, 2011 1:58 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] radical idea? I think that the main objection was not to the MPI_PROC_NULL semantics themselves, but the fact that querying and setting the state of a communicator is not thread safe. One thread can ask how many ranks are in a given state on comm A and another rank can then change the state of the comm by setting a rank to MPI_PROC_NULL, invalidating the results of the first thread's query. If we remove the ability to explicitly change the state of a rank, then notification can be handled in a more thread-safe way. [rich] So, again missing the context of the meeting, the functionality that I believe is needed is for an app to explicitly be able to recognize a failure has occurred, so that they can respond appropriately in the app. For point-to-point communications, I suppose it really does not matter if the library returns an error all the time, or if it returns success, if proc null semantics are used. When we move on to recovery, then the story will change - the library needs to know if it needs to restart communications - and this is intended to lead into that scenario. For collectives, I believe that forcing the app to recognize the failure before proceeding is essential - the app explicitly is acknowledging that collective ops in the new configuration, and can decide it to proceed based on the new state of the communicator. Our argument was that separate threads should not be working with the same comm at the same time as this doesn't work well currently (e.g. two threads trying to call collectives on the same comm at the same time). [rich] As for thread safety, this is not different than any other atomic operation - if you do things non-atomically, you get what you get. So here what is atomic is the" state data". In general, I think that if there was a way to do what we wanted without introducing 40? state quering/setting functions we would get less pushback. Getting rid of the MPI_PROC_NULL state opens up several options to such a simplification. [rich] Again, if the app does not mind ignoring errors, once a remote process has entered into the error state, this is ok. [[ personal opinion: I think MPI_PROC_NULL semantics is a nice touch, though it is a guess how much it would actually get used vs. immediately calling MPI_Comm_split or someday calling MPI_Comm_restore. I'd be willing to drop it if it allowed us to move forward. ]] [rich] I believe there are folks that will do both. With comm split, every one is getting a new rank (well, at least some, it depends on who fails). Some will want this, and others don't need it. So giving a broad option of choices is good. Rich Thanks, Dave -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Thursday, July 21, 2011 12:46 PM To: 'MPI 3.0 Fault Tolerance and Dynamic Process Control working Group' Subject: Re: [Mpi3-ft] radical idea? Since I missed the meeting - what were the objections people had to the proc null semantics ? Is the suggestion that the app explicitly deal with the missing rank all the time, or something else ? What was the motivation ? What were the problems they saw ? What were the alternative suggestions ? Rich Sent with Good (www . good . com) -----Original Message----- From: Howard Pritchard [mailto:[email protected]] Sent: Thursday, July 21, 2011 01:08 PM Eastern Standard Time To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] radical idea? Hi Darius, If we want to get something about RTS into MPI 3.0 I don't think we have time to manage it as a set of smaller proposals. If we can eliminate the state problem that bothered some at the last forum meeting that would be a good start. Also, if we could simplify the proposal some by removing the PROC_NULL semantics, I would be in favor of that. If we want to limit the use of RTS to a small number of use cases (like the NOAA example), then I could see deferring "repairable" communicators to 3.1. Howard Darius Buntinas wrote: > We could break the rts proposal into smaller ones: > > point-to-point: local up/down checks; errors on sending to failed processes > recognition/PROC_NULLification: Add a function to set a rank in a communicator to > MPI_PROC_NULL > fault-aware collectives: collectives don't hang, but they're permanently broken once a > proc in the communicator fails > "repairable" collectives: validate_all; collectives can be reactivated after failure > > I don't think anyone really objected to "point-to-point" or "fault-aware collectives". We'll have to work on the others. > > -d > > > On Jul 20, 2011, at 9:13 AM, Joshua Hursey wrote: > >> I'll have to think a bit more and come back to this thread. But I wanted to interject something I was thinking about on the plane ride back. What if we removed the notion of recognized failures? >> >> This was a point that was mentioned a couple times in discussion - that we have a bunch of functions and extra state on each communicator because we want to allow the application to recognize failures to get PROC_NULL semantics. If we remove the notion of recognized failures, then the up/down state on the group would be enough to track. So failed processes will always return an error regardless of if the failure has been 'seen' by the application before or not. >> >> The state of a process would be able to change as MPI finds out about new failures. But we can provide a 'state snapshot' object (which was mentioned in discussion, and I think is what Darius is getting at below) to allow for more consistent lookups if the application so desires. This removes the local/global list tracking on each handle, and moves it to a separate object that the user is in control of. The user can still reference the best known state if they are not concerned about consistency (e.g., MPI_Comm_validate_get_state(comm, ...) vs MPI_Snapshot_validate_get_state(snapshot_handle, ...)). >> >> Some applications would like the PROC_NULL semantics. But if we can convince ourselves that a library on top of MPI could provide those (by adding the proc_null check in the PMPI layer), then we might be able to reduce the complexity of the proposal by pushing some of the state tracking responsibility above MPI. >> >> I still have not figured out the implications on an application using collective operations if we remove the NULL state, but it is something to think about. >> >> -- Josh >> >> On Jul 19, 2011, at 6:11 PM, Darius Buntinas wrote: >> >>> The MPI_COMM_NULLIFY() function would effectively set the process to MPI_PROC_STATE_NULL. >>> >>> In the proposal we had MPI_PROC_STATE_NULL, _FAILED and _OK. I'm proposing separating NULL from FAILED and OK. So the MPI_COMM_GET_STATE() function (and friends) would let you query the (locally known) FAILED/OK state of the process, while MPI_COMM_NULLIFY() (and friends) would let you set the process to NULL. There would be essentially two state variables associated with each process: one indicating whether it's failed or not (let's call it LIVENESS), and the other whether it's has PROC_NULL semantics (call it NULLIFICATION). The LIVENESS state is controlled by the MPI library while the NULLIFICATION state is controlled by the user. The table below shows how these states would match up with the current proposal: >>> >>> Current proposal state LIVENESS NULLIFICATION >>> -----------------------+----------+--------------- >>> MPI_PROC_STATE_OK OK NORMAL >>> MPI_PROC_STATE_FAILED FAILED NORMAL >>> MPI_PROC_STATE_NULL FAILED NULL >>> <UNDEFINED> OK NULL >>> >>> Notice that there's a combination possible that's not covered by the current proposal. I'm not sure whether that's a useful state (or if we should disallow it). >>> >>> We'd could add a function to set the NULLIFICATION state from NULL to NORMAL for completeness. >>> >>> -d >>> >>> >>> On Jul 19, 2011, at 4:32 PM, Solt, David George wrote: >>> >>>> This works for "reading" state, but has no way to set a processes state. (Not sure how radical your trying to go here... is part of the change here that there would no longer be a MPI_PROC_STATE_NULL state?) >>>> Dave >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Darius Buntinas >>>> Sent: Tuesday, July 19, 2011 3:17 PM >>>> To: Darius Buntinas >>>> Cc: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] radical idea? >>>> >>>> >>>> Howard pointed out that I forgot to add a FREE operation: >>>> >>>> MPI_PROC_STATE_FREE(state_handle) >>>> INOUT: MPI_PROC_STATE state_handle >>>> >>>> -d >>>> >>>> On Jul 19, 2011, at 3:07 PM, Darius Buntinas wrote: >>>> >>>>> MPI_COMM_GET_STATE(comm, state_handle) >>>>> IN: MPI_COMM comm >>>>> OUT: MPI_PROC_STATE state_handle >>>>> and ditto for GROUP, FILE, WIN as necessary >>>>> >>>>> MPI_GET_PROC_STATE_SIZE(state_handle, mask, size) >>>>> IN: MPI_PROC_STATE state_handle >>>>> IN: int mask >>>>> OUT: int size >>>>> >>>>> MPI_GET_PROC_STATE_LIST(state_handle, mask, list) >>>>> IN: MPI_PROC_STATE state_handle >>>>> IN: int mask >>>>> OUT: int list[] >>>>> >>>>> MPI_GET_PROC_STATE_NEW(state_handle1, state_handle2, state_handle_new) >>>>> IN: MPI_PROC_STATE state_handle1 >>>>> IN: MPI_PROC_STATE state_handle2 >>>>> OUT: MPI_PROC_STATE state_handle_new >>>>> This gives newly failed processes in state_handle2 since state_handle1. >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Howard Pritchard Software Engineering Cray, Inc. _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org hxxp://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0