July 2011 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] radical idea?
by Darius Buntinas 19 Jul '11

19 Jul '11

Here are some ideas on how we can change the interface based on the feedback we got. I haven't thought too deeply on all the implications of these, but it's a starting point for discussion. -d Let's not call (local) VALIDATE VALIDATE and, let's split out UP/DOWN from RANK_NULLification. Then to query for UP/DOWN, use a handle to an explicit "state" object (as opposed to an implicit "snapshot"), then query that. e.g.: MPI_COMM_GET_STATE(comm, state_handle) IN: MPI_COMM comm OUT: MPI_PROC_STATE state_handle and ditto for GROUP, FILE, WIN as necessary MPI_GET_PROC_STATE_SIZE(state_handle, mask, size) IN: MPI_PROC_STATE state_handle IN: int mask OUT: int size MPI_GET_PROC_STATE_LIST(state_handle, mask, list) IN: MPI_PROC_STATE state_handle IN: int mask OUT: int list[] MPI_GET_PROC_STATE_NEW(state_handle1, state_handle2, state_handle_new) IN: MPI_PROC_STATE state_handle1 IN: MPI_PROC_STATE state_handle2 OUT: MPI_PROC_STATE state_handle_new This gives newly failed processes in state_handle2 since state_handle1. This addresses issues people had with different threads calling VALIDATE and resetting the "new" flag for other threads. We can then have a MPI_COMM_NULLIFY() function (or whatever we decide to call it), that would effectively set the rank to MPI_RANK_NULL: MPI_COMM_NULLIFY(comm, rank) IN: MPI_COMM comm IN: int rank MPI_COMM_NULLIFY_STATE(comm, mask, state_handle) This sets all ranks described by mask and state_handle to PROC_NULL MPI_COMM_NULLIFY_GROUP(comm, group) Set all procs in group to PROC_NULL in comm. Same as logically doing: foreach p in group MPI_COMM_NULLIFY(comm, rank-of-p-in-comm) The operations would be idempotent, and can be called on either live or failed processes. Note, a process can be (UP or DOWN) x (NORMAL or NULL). VALIDATE_ALL can be renamed to VALIDATE. It returns a state_handle that can be queried for failed processes. Then we can describe it as having the effect of deciding on a common set of failed processes across the comm, setting state_handle to that set, and calling MPI_COMM_NULLIFY() on each failed process in that set. MPI_COMM_VALIDATE(comm, state_handle) IN: MPI_COMM comm OUT: MPI_PROC_STATE state_handle We may also want a function to "link" one comm's PROC_NULLified state with another, so that if comm_A and comm_B are linked, calling MPI_COMM_NULLIFY on comm_A also NULLifies it on comm_B. We can have a restriction that comm_B is a subset of comm_A or vv.

1 0

[Mpi3-ft] More notes from jul19 mpi forum discussion
by Darius Buntinas 19 Jul '11

19 Jul '11

Here are some notes from today's discussion. -d In group_validate_get_state, process states are defined in 17.4 not 17.4.1. Also flags are mentioned in 17.4.1, so both sections should be referenced. Add text that the user cannot use the group routines to change the rank state to NULL SET_STATE_NULL: add a mask parameter so one does not need to allocate a rank_infos array to set all to NULL MPI_COMM_KILL: require comm to be an intracomm? Order of reduction operations: There's something in the standard (advice to user possibly) that says that the order of evaluation is not defined, but it will not change from one call to the other. However, it's possible that the order may change after a validate_all because the tree has changed. We should possibly include an advice to users on this in the FT chapter.

1 0

[Mpi3-ft] Some remarks from the presentation at the MPI forum
by Aurélien Bouteiller 19 Jul '11

19 Jul '11

Here are some of the informal comments I'd like to make regarding the 274 proposition presented at the MPI forum. == Why do you propose duplicate functions for groups and communicator operations ? Some proposed to remove the comms, but as Josh correctly stated, the fact that it is attached on a per-communicator basis is a key feature to enable software layer composition. On the other end, I don't see a clear identified need for group operations. The only thing that is enabled by group operations is to validate a group with which you cannot communicate (no communicator with it). Also, group operations are not homogenous with the rest, they don't have a set (because that would be harmful). I think these should be removed from the proposal, except if some clear need is identified. == Section 17.6.2::19 should be linked from the Finalize definition, where the example of a code doing IO on rank 0 after finalize is presented. That would keep all FT related considerations in the FT chapter, at the exception of a warning to users that could read like "beware that when considering a fault tolerant application, rank 0 might be dead, in which case the result of finalize is defined in 17.6.2." == Why the ordering on lowest rank return code if rank 0 is dead ? The purpose is unclear, but the cost on the implementation of Finalize is real. We could just specify that the return value from any of the surviving processes (if any) can be taken. It should define what happens when nobody calls abort, rank 0 is dead, and finalize does not return on ranks>0 (valid by the standard). == MPI_Kill: might go in another ticket about soft errors. The current ticket is about fail-stop, this is the only thing about soft errors, hence I see it to be out of scope. It is a polarizing controversy, that might need significant rework in the broader context of byzantine errors. I advocate it is postponed. == Ordering of reduce operations may change across ranks, after Validate_all recognized a failure, and more generally collective trees are modified. That's all folks! aurelien -- * Dr. Aurï¿½lien Bouteiller * Research Scientist at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 350 * Knoxville, TN 37996 * 865 974 6321

1 0

[Mpi3-ft] Notes from the reading
by Darius Buntinas 19 Jul '11

19 Jul '11

Here are the notes I took at the first reading on Monday. They're rough, but I can clarify anything as needed. -d MPI_ERR_RANK_FAILED --> MPI_ERR_PROC_FAILED "is failed" --> "has failed" p346l39 "the call the" --> "the call to" Example 8.7, add something like "assuming the above situation doesn't happen" p346 l39: after MPI_FINALIZE, add something like "but all processes have not been aborted" MPI_Rank_info --> MPI_Rank_state p567 l38: Pull out "fault" as it's own definition item fail-stop process failure definition: remove "often due to a component crash" clarify failed-stop failure: stops wrt MPI, stops communicating, stops responding alive state: remove "normal", or define it as "not failed" remove references to state, or add forward references recognized/unrecognized failed processes: add something like "recognized with the validate function" collectively active: need to be recognized using validate_all p568 l22: change to something like "all processes will eventually be able to know of any failed process" Section 17.4: Mention that state is bound to an abject.

1 0

Re: [Mpi3-ft] RTS Slides: First Draft
by George R Carr Jr 15 Jul '11

15 Jul '11

>Attached is the latest revision of the slides that will be presented >on Monday. The 3rd application group could not get their slides >approved in time, so we have slides from 2 applications which should >be ok. > >If you have any comments/questions/edits let me know. Otherwise I'll >see folks on Monday. > >Thanks, >Josh Looks good to me. I'll be there is any tough questions come up. > > >On Thu, Jul 7, 2011 at 11:12 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> Attached is a PDF version of the slides for the MPI Forum meeting. The >> application slides are just place holders for now - content will be >> available for those slides next week. >> >> Let me know what you think, and if you have any feedback on the slides. >> >> Thanks, >> Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> > > > >-- >Joshua Hursey >Postdoctoral Research Associate >Oak Ridge National Laboratory >http://users.nccs.gov/~jjhursey > >Content-Type: application/pdf; name="2011-07-RTS.pdf" >Content-Disposition: attachment; filename="2011-07-RTS.pdf" >X-Attachment-Id: f_gq5isc7o1 > >Attachment converted: hiroshige:2011-07-RTS.pdf (PDF /ï¿½ICï¿½) (0030D637) >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- George R Carr Jr U.S. Department of Commerce NOAA/OAR/GSD, DSRC 2B142 Tel: 303-497-4714 Email: George.Carr(a)noaa.gov

1 0

Re: [Mpi3-ft] RTS Slides: First Draft
by Josh Hursey 15 Jul '11

15 Jul '11

Attached is the latest revision of the slides that will be presented on Monday. The 3rd application group could not get their slides approved in time, so we have slides from 2 applications which should be ok. If you have any comments/questions/edits let me know. Otherwise I'll see folks on Monday. Thanks, Josh On Thu, Jul 7, 2011 at 11:12 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Attached is a PDF version of the slides for the MPI Forum meeting. The > application slides are just place holders for now - content will be > available for those slides next week. > > Let me know what you think, and if you have any feedback on the slides. > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] My powerpoint attached
by George R Carr Jr 07 Jul '11

07 Jul '11

Please excuse my broadcast. The email was meant for Josh. -- George R Carr Jr U.S. Department of Commerce NOAA/OAR/GSD, DSRC 2B142 Tel: 303-497-4714 Email: George.Carr(a)noaa.gov

1 0

[Mpi3-ft] My powerpoint attached
by George R Carr Jr 07 Jul '11

07 Jul '11

I've attached my working powerpoints slides. Should we find time to talk about these? Regards, George >Attached is a PDF version of the slides for the MPI Forum meeting. The >application slides are just place holders for now - content will be >available for those slides next week. > >Let me know what you think, and if you have any feedback on the slides. > >Thanks, >Josh > >-- >Joshua Hursey >Postdoctoral Research Associate >Oak Ridge National Laboratory >http://users.nccs.gov/~jjhursey > >Content-Type: application/pdf; name="2011-07-RTS.pdf" >Content-Disposition: attachment; filename="2011-07-RTS.pdf" >X-Attachment-Id: f_gptuv1vz0 > >Attachment converted: hiroshige:2011-07-RTS.pdf (PDF /«IC») (002E419C) >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- George R Carr Jr U.S. Department of Commerce NOAA/OAR/GSD, DSRC 2B142 Tel: 303-497-4714 Email: George.Carr(a)noaa.gov

1 0

[Mpi3-ft] RTS Slides: First Draft
by Josh Hursey 07 Jul '11

07 Jul '11

Attached is a PDF version of the slides for the MPI Forum meeting. The application slides are just place holders for now - content will be available for those slides next week. Let me know what you think, and if you have any feedback on the slides. Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT WG Teleconference Today @ Noon EDT
by Josh Hursey 06 Jul '11

06 Jul '11

Reminder that we have a teleconf today, Wednesday 7/6/2011, at Noon EDT. Date: July 6, 2011 Time: Noon EDT/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - FT Subversion Repository Re-organization (Darius B.) - Run-through stabilization proposal Presentation Prep (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Other items? Additional Agenda Items [if time permits]: - Piggybacking? (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Process recovery proposal (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * July 6, 2011 (Teleconf) * July 18, 2011 (MPI Forum Meeting) * Aug. 3, 2011 (Teleconf) * Aug. 17, 2011 (Teleconf) * Aug. 31, 2011 (Teleconf) * Sept. 14, 2011 (Teleconf) * Sept. 22, 2011 (MPI Forum Meeting) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0