January 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization
by George Bosilca 18 Jan '12

18 Jan '12

On Jan 18, 2012, at 14:06 , Rolf Rabenseifner wrote: > To make MPI_Failhandler_set_mode collective is another choice. > Collective over what? All connected processes? As you pointed out, collective over what? Per communicator? Per MPI_COMM_WORLD? We should forget about the all connected processes, as the current MPI standard have a very careless definition of it. > What is with processes that are spawned after MPI_Failhandler_set_mode? I don't think this function is needed at all. The setting of the failure handler (MPI_Comm_set_failhandler) should be collective over the communicator provided as argument. In the case of spawned or connected processes, it is the upper layer responsibility to set the failure handler to what make sense for the application stack (either by creating an intra-comm and setting the failure handler there or we might want to extend the logic of the MPI_Comm_set_failhandler to cover inter-comms). > The other idea is, to allow setting special options before MPI_Init/MPI_Init_thread. I think it makes more sense if these options are somehow global … at least to the MPI_COMM_WORLD. Maybe via the command line? > The only requirement is, that the FT proposal is consistent > with the rest of MPI. We already have some inconsistencies in the current standard, it will be abusing to require from the FT proposal what the remaining of the MPI standard fails to deliver. Don't get me wrong here, I'm not saying we should let it loose either. Some level of decency should be enforced. george. > Best regards > Rolf > > ----- Original Message ----- >> From: "George Bosilca" <bosilca(a)eecs.utk.edu> >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> >> Cc: "Terry D. Dontje" <terry.dontje(a)oracle.com>, "Josh Hursey" <jjhursey(a)open-mpi.org>, "Rolf Rabenseifner" >> <rabenseifner(a)hlrs.de>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, "Pavan Balaji" <balaji(a)mcs.anl.gov> >> Sent: Wednesday, January 18, 2012 4:26:43 PM >> Subject: Re: MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization >> I concur with the previous statements. As Rolf highlighted it in his >> email, one of the reasons of this new proposal is to fix the "unclear" >> collective behavior of MPI_Failhandler_set_mode. I don't see the >> unclearness, and here are two of my reasons. >> >> >> 1. There is no reason to have such a function >> (MPI_Failhandler_set_mode), setting of the fail handler should ALWAYS >> be collective, otherwise the entire purpose of the fail handler is >> annihilated. >> >> >> 2. If no collective behavior is required (meaning the software stack >> doesn't have to be rebuild in a collective way), then the fail handler >> is a clear overkill. A saner and more clear behavior can be obtained >> by using local Error handler with carefully crafted requests (as an >> example a non-blocking, never to be matched, request on a duplicate of >> MPI_COMM_WORLD can do the trick). >> >> >> george. >> >> >> >> On Jan 18, 2012, at 09:56 , Josh Hursey wrote: >> >> >> I like the motivation of the proposal, but I think Terry has a good >> point. It seems a bit like a hack to repurpose the required/provided >> arguments to achieve semantic assertions. I would almost prefer some >> other functionality that must be called before MPI_Init{_thread} that >> would explicitly set these options. That starts to sound like the >> assertion ticket that Terry mentioned. So maybe they can be merged or >> revised. >> >> >> I am also a bit concerned about having conditional semantics in the >> MPI standard. Though the FT proposal is founded in the condition that >> the semantics are only meaningful when the error handler is not >> ARE_FATAL, which is conditional. So I am a bit torn on this point. >> >> >> One thing that your proposal should clearly specify is whether the >> specified bits must be set to the same value at all processes/threads. >> Additionally, what if two MPI_COMM_WORLDs connect/accept but have >> different bits set? Does that restrict how these two world can >> interact? This was one of the problems posed for the >> Failhandler_set_mode() semantics that still needs to be addressed >> here, but in a more general sense. >> >> >> So I think it is an interesting proposal worth considering further. >> Setting options like 'enable FT' at initialization time (or just >> before initialization time) might allow the MPI implementation to >> optimize the library appropriately during setup (choosing different >> components or algorithms). It might be worth looking at the assertions >> proposal to see if there is a viable alternative solution there that >> would achieve the same goals as this proposal without repurposing the >> required/provided arguments of MPI_Init_thread. >> >> >> -- Josh >> >> >> >> On Wed, Jan 18, 2012 at 9:32 AM, TERRY DONTJE < >> terry.dontje(a)oracle.com > wrote: >> >> >> >> I think the idea is worthwhile but it really smells similar to the >> defunct assertion ticket. I really find piggy-backing the ft, cancel >> and any_source modes onto the required/provided bits a little >> unpleasing to my senses. The reason I am displeased with the proposal >> is it seems to slightly open a door to give an application the ability >> to give hints and if we are going to do that we might as well open the >> door fully and allow vendor specific hints. Doing the latter will >> require more than the require/provided bits. >> >> The above aside, if the proposal is passed I guess my only other >> comment is the of moving the MPI_INIT_THREAD & MPI_QUERY_THREAD to 8.7 >> (startup) seems odd to me. I guess I can see the reasoning of moving >> the interface to the Startup section but then the threadsafety portion >> of 12.4.3 section seems to stick out strangely IMO. >> >> --td >> >> >> >> On 1/17/2012 5:31 AM, Rolf Rabenseifner wrote: >> >> Dear committees of - FT, - MPI_Init --> 8. Environmental Management, - >> MPI_Init_thread --> 12. External Interfaces. Before discussing >> details, I would like to get a clear answer whether you believe that >> the proposal below is a good or bad idea. As already mentioned at the >> Jan. 2012 meeting, I would like to propose that the FT group may >> substitute the unclear collective behavior of MPI_Failhandler_set_mode >> by adding the mode to the MPI initialization. For this, I added a >> proposal to slide 4 in >> MPI_Forum_Overview_MPI-3.0_Jan2012_action-items.ppt (see my previous >> mail to the MPI-Forum list) - If appropriate, a new ticket that >> enhances MPI_INIT_THREAD -- Required and provided as “bit vector” of >> "bit-wise OR" of required_/provided_threadsafety | >> required_/provided_ft_mode | required_/provided_cancel_mode | >> required_/provided_any_source_mode -- New mask-constants >> MPI_THREAD_MASK, MPI_FT_MASK, MPI_CANCEL_MASK, MPI_ANY_SOURCE_MASK -- >> With existing values for required_/provided_threadsafety >> MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, MPI_THREAD_SERIALIZED, and >> MPI_THREAD_MULTIPLE. -- With new values for - >> required_/provided_ft_mode = MPI_FT_NONE=0, or >> MPI_FT_FAILHANDLER_MODE_ALL≠0, or MPI_FT_FAILHANDLER_MODE_SUBSET≠0, or >> - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED=0, or >> MPI_NO_CANCEL≠0 - required_/provided_any_source_mode = >> MPI_ANY_SOURCE_ALLOWED=0, or MPI_NO_ANY_SOURCE≠0 -- Values must be set >> identical for all processes in an MPI_COMM_WORLD - It is easier to >> relax about this in further versions of MPI than to relax already now >> and to restrict later as now done in ticket #222 for >> required_/provided_threadsafety - For each of the for "variables", a >> different decision can be done. -- At least for >> required_/provided_cancel_mode and ...any_source_mode, I would require >> that the provided value must be identical to the required value. >> Reason: Internally, the value can be ignored. -- For >> required_/provided_ft_mode, I would recommend to allow that >> provided_ft_mode must be - identical to required_ft_mode or >> MPI_FT_NONE - and the same in all processes. -- MPI_INIT_THREAD & >> MPI_QUERY_THREAD moves - from 12.4.3 (External Interfaces) - to 8.7 >> (Startup) - but explanations to ...threadsafety | ...ft_mode | >> ...cancel_mode | ...any_source_mode are kept or written in the >> appropriate sections 12.4.3, new 17.5 (FT Environm.), 3.8.4 (Cancel), >> 3.2.4 (Blocking Receive) -- A call to MPI_INIT is identical to >> MPI_INIT_THREAD with - the rules in 12.4.3 about >> required_/provided_threadsafety - required_/provided_ft_mode = >> MPI_FT_NONE, - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED, - >> required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED -- This >> ticket would have the following properties: - It is clearly >> source-code and ABI backward compatible, because - the values of >> MPI_THREAD_SINGLE, _FUNNELED, ... need not to be changed and >> MPI_THREAD_SINGLE need not to be zero; - the values representing the >> current MPI-2.2 quality are set to zero: MPI_FT_NONE=0, >> MPI_CANCEL_ALLOWED=0, and MPI_ANY_SOURCE_ALLOWED=0 - It is very >> unlikely that an implementation has used more than 24 different bits >> in these 4 integer constants MPI_THREAD_SINGLE, ... _MULTIPLE. >> Therefore MPI_THREAD_MASK would have a maximum of 24 bits. Enough room >> for the 4 bits needed together for the other three ..._MASKs. - FT can >> be switched on or off at the MPI initialization and is switched off in >> unchanged applications. Therefore no backward-compatibility-problem >> with a modified behavior of the default error handlers when FT is >> switched on. - Normally there should be enough bit-space for further >> decisions at MPI initialization. - The decisions about the cancel and >> any_source values would be done in different tickets - FT quality is >> optional if we add the rule that provided_ft_mode may be identical to >> required_ft_mode ***or*** MPI_FT_NONE - This rule can be changed in a >> further version of MPI without backward-compatibility-problems. I >> would like to get a reply from - the FT group - the chapter committee >> of MPI_Init --> 8. Environmental Manag. George Bosilca(c), Josh >> Hursey, Terry Dontje - the chapter committee of MPI_Init_thread --> >> 12. External Interf. Bronis R. de Supinski(c), Pavan Balaji Before >> discussing details, I would like to get a clear answer whether you >> believe that this is a good or bad idea. Best regards Rolf >> >> -- >> >> <Mail Attachment.gif> >> >> >> >> >> Terry D. Dontje | Principal Software Engineer >> >> Developer Tools Engineering | +1.781.442.2631 >> Oracle - Performance Technologies >> 95 Network Drive, Burlington, MA 01803 >> Email terry.dontje(a)oracle.com >> >> >> >> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [Mpi3-ft] MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization
by Rolf Rabenseifner 18 Jan '12

18 Jan '12

To make MPI_Failhandler_set_mode collective is another choice. Collective over what? All connected processes? What is with processes that are spawned after MPI_Failhandler_set_mode? The other idea is, to allow setting special options before MPI_Init/MPI_Init_thread. The only requirement is, that the FT proposal is consistent with the rest of MPI. Best regards Rolf ----- Original Message ----- > From: "George Bosilca" <bosilca(a)eecs.utk.edu> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> > Cc: "Terry D. Dontje" <terry.dontje(a)oracle.com>, "Josh Hursey" <jjhursey(a)open-mpi.org>, "Rolf Rabenseifner" > <rabenseifner(a)hlrs.de>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, "Pavan Balaji" <balaji(a)mcs.anl.gov> > Sent: Wednesday, January 18, 2012 4:26:43 PM > Subject: Re: MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization > I concur with the previous statements. As Rolf highlighted it in his > email, one of the reasons of this new proposal is to fix the "unclear" > collective behavior of MPI_Failhandler_set_mode. I don't see the > unclearness, and here are two of my reasons. > > > 1. There is no reason to have such a function > (MPI_Failhandler_set_mode), setting of the fail handler should ALWAYS > be collective, otherwise the entire purpose of the fail handler is > annihilated. > > > 2. If no collective behavior is required (meaning the software stack > doesn't have to be rebuild in a collective way), then the fail handler > is a clear overkill. A saner and more clear behavior can be obtained > by using local Error handler with carefully crafted requests (as an > example a non-blocking, never to be matched, request on a duplicate of > MPI_COMM_WORLD can do the trick). > > > george. > > > > On Jan 18, 2012, at 09:56 , Josh Hursey wrote: > > > I like the motivation of the proposal, but I think Terry has a good > point. It seems a bit like a hack to repurpose the required/provided > arguments to achieve semantic assertions. I would almost prefer some > other functionality that must be called before MPI_Init{_thread} that > would explicitly set these options. That starts to sound like the > assertion ticket that Terry mentioned. So maybe they can be merged or > revised. > > > I am also a bit concerned about having conditional semantics in the > MPI standard. Though the FT proposal is founded in the condition that > the semantics are only meaningful when the error handler is not > ARE_FATAL, which is conditional. So I am a bit torn on this point. > > > One thing that your proposal should clearly specify is whether the > specified bits must be set to the same value at all processes/threads. > Additionally, what if two MPI_COMM_WORLDs connect/accept but have > different bits set? Does that restrict how these two world can > interact? This was one of the problems posed for the > Failhandler_set_mode() semantics that still needs to be addressed > here, but in a more general sense. > > > So I think it is an interesting proposal worth considering further. > Setting options like 'enable FT' at initialization time (or just > before initialization time) might allow the MPI implementation to > optimize the library appropriately during setup (choosing different > components or algorithms). It might be worth looking at the assertions > proposal to see if there is a viable alternative solution there that > would achieve the same goals as this proposal without repurposing the > required/provided arguments of MPI_Init_thread. > > > -- Josh > > > > On Wed, Jan 18, 2012 at 9:32 AM, TERRY DONTJE < > terry.dontje(a)oracle.com > wrote: > > > > I think the idea is worthwhile but it really smells similar to the > defunct assertion ticket. I really find piggy-backing the ft, cancel > and any_source modes onto the required/provided bits a little > unpleasing to my senses. The reason I am displeased with the proposal > is it seems to slightly open a door to give an application the ability > to give hints and if we are going to do that we might as well open the > door fully and allow vendor specific hints. Doing the latter will > require more than the require/provided bits. > > The above aside, if the proposal is passed I guess my only other > comment is the of moving the MPI_INIT_THREAD & MPI_QUERY_THREAD to 8.7 > (startup) seems odd to me. I guess I can see the reasoning of moving > the interface to the Startup section but then the threadsafety portion > of 12.4.3 section seems to stick out strangely IMO. > > --td > > > > On 1/17/2012 5:31 AM, Rolf Rabenseifner wrote: > > Dear committees of - FT, - MPI_Init --> 8. Environmental Management, - > MPI_Init_thread --> 12. External Interfaces. Before discussing > details, I would like to get a clear answer whether you believe that > the proposal below is a good or bad idea. As already mentioned at the > Jan. 2012 meeting, I would like to propose that the FT group may > substitute the unclear collective behavior of MPI_Failhandler_set_mode > by adding the mode to the MPI initialization. For this, I added a > proposal to slide 4 in > MPI_Forum_Overview_MPI-3.0_Jan2012_action-items.ppt (see my previous > mail to the MPI-Forum list) - If appropriate, a new ticket that > enhances MPI_INIT_THREAD -- Required and provided as “bit vector” of > "bit-wise OR" of required_/provided_threadsafety | > required_/provided_ft_mode | required_/provided_cancel_mode | > required_/provided_any_source_mode -- New mask-constants > MPI_THREAD_MASK, MPI_FT_MASK, MPI_CANCEL_MASK, MPI_ANY_SOURCE_MASK -- > With existing values for required_/provided_threadsafety > MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, MPI_THREAD_SERIALIZED, and > MPI_THREAD_MULTIPLE. -- With new values for - > required_/provided_ft_mode = MPI_FT_NONE=0, or > MPI_FT_FAILHANDLER_MODE_ALL≠0, or MPI_FT_FAILHANDLER_MODE_SUBSET≠0, or > - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED=0, or > MPI_NO_CANCEL≠0 - required_/provided_any_source_mode = > MPI_ANY_SOURCE_ALLOWED=0, or MPI_NO_ANY_SOURCE≠0 -- Values must be set > identical for all processes in an MPI_COMM_WORLD - It is easier to > relax about this in further versions of MPI than to relax already now > and to restrict later as now done in ticket #222 for > required_/provided_threadsafety - For each of the for "variables", a > different decision can be done. -- At least for > required_/provided_cancel_mode and ...any_source_mode, I would require > that the provided value must be identical to the required value. > Reason: Internally, the value can be ignored. -- For > required_/provided_ft_mode, I would recommend to allow that > provided_ft_mode must be - identical to required_ft_mode or > MPI_FT_NONE - and the same in all processes. -- MPI_INIT_THREAD & > MPI_QUERY_THREAD moves - from 12.4.3 (External Interfaces) - to 8.7 > (Startup) - but explanations to ...threadsafety | ...ft_mode | > ...cancel_mode | ...any_source_mode are kept or written in the > appropriate sections 12.4.3, new 17.5 (FT Environm.), 3.8.4 (Cancel), > 3.2.4 (Blocking Receive) -- A call to MPI_INIT is identical to > MPI_INIT_THREAD with - the rules in 12.4.3 about > required_/provided_threadsafety - required_/provided_ft_mode = > MPI_FT_NONE, - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED, - > required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED -- This > ticket would have the following properties: - It is clearly > source-code and ABI backward compatible, because - the values of > MPI_THREAD_SINGLE, _FUNNELED, ... need not to be changed and > MPI_THREAD_SINGLE need not to be zero; - the values representing the > current MPI-2.2 quality are set to zero: MPI_FT_NONE=0, > MPI_CANCEL_ALLOWED=0, and MPI_ANY_SOURCE_ALLOWED=0 - It is very > unlikely that an implementation has used more than 24 different bits > in these 4 integer constants MPI_THREAD_SINGLE, ... _MULTIPLE. > Therefore MPI_THREAD_MASK would have a maximum of 24 bits. Enough room > for the 4 bits needed together for the other three ..._MASKs. - FT can > be switched on or off at the MPI initialization and is switched off in > unchanged applications. Therefore no backward-compatibility-problem > with a modified behavior of the default error handlers when FT is > switched on. - Normally there should be enough bit-space for further > decisions at MPI initialization. - The decisions about the cancel and > any_source values would be done in different tickets - FT quality is > optional if we add the rule that provided_ft_mode may be identical to > required_ft_mode ***or*** MPI_FT_NONE - This rule can be changed in a > further version of MPI without backward-compatibility-problems. I > would like to get a reply from - the FT group - the chapter committee > of MPI_Init --> 8. Environmental Manag. George Bosilca(c), Josh > Hursey, Terry Dontje - the chapter committee of MPI_Init_thread --> > 12. External Interf. Bronis R. de Supinski(c), Pavan Balaji Before > discussing details, I would like to get a clear answer whether you > believe that this is a good or bad idea. Best regards Rolf > > -- > > <Mail Attachment.gif> > > > > > Terry D. Dontje | Principal Software Engineer > > Developer Tools Engineering | +1.781.442.2631 > Oracle - Performance Technologies > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > > > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [Mpi3-ft] MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization
by George Bosilca 18 Jan '12

18 Jan '12

I concur with the previous statements. As Rolf highlighted it in his email, one of the reasons of this new proposal is to fix the "unclear" collective behavior of MPI_Failhandler_set_mode. I don't see the unclearness, and here are two of my reasons. 1. There is no reason to have such a function (MPI_Failhandler_set_mode), setting of the fail handler should ALWAYS be collective, otherwise the entire purpose of the fail handler is annihilated. 2. If no collective behavior is required (meaning the software stack doesn't have to be rebuild in a collective way), then the fail handler is a clear overkill. A saner and more clear behavior can be obtained by using local Error handler with carefully crafted requests (as an example a non-blocking, never to be matched, request on a duplicate of MPI_COMM_WORLD can do the trick). george. On Jan 18, 2012, at 09:56 , Josh Hursey wrote: > I like the motivation of the proposal, but I think Terry has a good point. It seems a bit like a hack to repurpose the required/provided arguments to achieve semantic assertions. I would almost prefer some other functionality that must be called before MPI_Init{_thread} that would explicitly set these options. That starts to sound like the assertion ticket that Terry mentioned. So maybe they can be merged or revised. > > I am also a bit concerned about having conditional semantics in the MPI standard. Though the FT proposal is founded in the condition that the semantics are only meaningful when the error handler is not ARE_FATAL, which is conditional. So I am a bit torn on this point. > > One thing that your proposal should clearly specify is whether the specified bits must be set to the same value at all processes/threads. Additionally, what if two MPI_COMM_WORLDs connect/accept but have different bits set? Does that restrict how these two world can interact? This was one of the problems posed for the Failhandler_set_mode() semantics that still needs to be addressed here, but in a more general sense. > > So I think it is an interesting proposal worth considering further. Setting options like 'enable FT' at initialization time (or just before initialization time) might allow the MPI implementation to optimize the library appropriately during setup (choosing different components or algorithms). It might be worth looking at the assertions proposal to see if there is a viable alternative solution there that would achieve the same goals as this proposal without repurposing the required/provided arguments of MPI_Init_thread. > > -- Josh > > > On Wed, Jan 18, 2012 at 9:32 AM, TERRY DONTJE <terry.dontje(a)oracle.com> wrote: > I think the idea is worthwhile but it really smells similar to the defunct assertion ticket. I really find piggy-backing the ft, cancel and any_source modes onto the required/provided bits a little unpleasing to my senses. The reason I am displeased with the proposal is it seems to slightly open a door to give an application the ability to give hints and if we are going to do that we might as well open the door fully and allow vendor specific hints. Doing the latter will require more than the require/provided bits. > > The above aside, if the proposal is passed I guess my only other comment is the of moving the MPI_INIT_THREAD & MPI_QUERY_THREAD to 8.7 (startup) seems odd to me. I guess I can see the reasoning of moving the interface to the Startup section but then the threadsafety portion of 12.4.3 section seems to stick out strangely IMO. > > --td > > > On 1/17/2012 5:31 AM, Rolf Rabenseifner wrote: >> >> Dear committees of >> - FT, >> - MPI_Init --> 8. Environmental Management, >> - MPI_Init_thread --> 12. External Interfaces. >> >> Before discussing details, I would like to get a clear answer >> whether you believe that the proposal below is a good or bad idea. >> >> As already mentioned at the Jan. 2012 meeting, I would like to propose >> that the FT group may substitute the unclear collective behavior >> of MPI_Failhandler_set_mode by adding the mode to the MPI initialization. >> >> For this, I added a proposal to slide 4 in >> MPI_Forum_Overview_MPI-3.0_Jan2012_action-items.ppt >> (see my previous mail to the MPI-Forum list) >> >> - If appropriate, a new ticket that enhances MPI_INIT_THREAD >> -- Required and provided as “bit vector” of "bit-wise OR" of >> required_/provided_threadsafety >> | required_/provided_ft_mode >> | required_/provided_cancel_mode >> | required_/provided_any_source_mode >> -- New mask-constants MPI_THREAD_MASK, MPI_FT_MASK, >> MPI_CANCEL_MASK, MPI_ANY_SOURCE_MASK >> -- With existing values for required_/provided_threadsafety >> MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, >> MPI_THREAD_SERIALIZED, and MPI_THREAD_MULTIPLE. >> -- With new values for >> - required_/provided_ft_mode = >> MPI_FT_NONE=0, or >> MPI_FT_FAILHANDLER_MODE_ALL≠0, or >> MPI_FT_FAILHANDLER_MODE_SUBSET≠0, or >> - required_/provided_cancel_mode = >> MPI_CANCEL_ALLOWED=0, or >> MPI_NO_CANCEL≠0 >> - required_/provided_any_source_mode = >> MPI_ANY_SOURCE_ALLOWED=0, or >> MPI_NO_ANY_SOURCE≠0 >> -- Values must be set identical for all processes in an MPI_COMM_WORLD >> - It is easier to relax about this in further versions of MPI >> than to relax already now and to restrict later >> as now done in ticket #222 for required_/provided_threadsafety >> - For each of the for "variables", a different decision >> can be done. >> -- At least for required_/provided_cancel_mode and >> ...any_source_mode, I would require that the provided >> value must be identical to the required value. >> Reason: Internally, the value can be ignored. >> -- For required_/provided_ft_mode, I would recommend >> to allow that provided_ft_mode must be >> - identical to required_ft_mode or MPI_FT_NONE >> - and the same in all processes. >> -- MPI_INIT_THREAD & MPI_QUERY_THREAD moves >> - from 12.4.3 (External Interfaces) >> - to 8.7 (Startup) >> - but explanations to ...threadsafety | ...ft_mode | >> ...cancel_mode | ...any_source_mode are kept or written >> in the appropriate sections 12.4.3, new 17.5 (FT Environm.), >> 3.8.4 (Cancel), 3.2.4 (Blocking Receive) >> -- A call to MPI_INIT is identical to MPI_INIT_THREAD with >> - the rules in 12.4.3 about required_/provided_threadsafety >> - required_/provided_ft_mode = MPI_FT_NONE, >> - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED, >> - required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED >> -- This ticket would have the following properties: >> - It is clearly source-code and ABI backward compatible, >> because >> - the values of MPI_THREAD_SINGLE, _FUNNELED, ... >> need not to be changed and MPI_THREAD_SINGLE need >> not to be zero; >> - the values representing the current MPI-2.2 quality >> are set to zero: MPI_FT_NONE=0, >> MPI_CANCEL_ALLOWED=0, and MPI_ANY_SOURCE_ALLOWED=0 >> - It is very unlikely that an implementation has used >> more than 24 different bits in these 4 integer >> constants MPI_THREAD_SINGLE, ... _MULTIPLE. >> Therefore MPI_THREAD_MASK would have a maximum >> of 24 bits. Enough room for the 4 bits needed >> together for the other three ..._MASKs. >> - FT can be switched on or off at the MPI initialization >> and is switched off in unchanged applications. >> Therefore no backward-compatibility-problem with a >> modified behavior of the default error handlers >> when FT is switched on. >> - Normally there should be enough bit-space for further >> decisions at MPI initialization. >> - The decisions about the cancel and any_source >> values would be done in different tickets >> - FT quality is optional if we add the >> rule that provided_ft_mode may be identical >> to required_ft_mode ***or*** MPI_FT_NONE >> - This rule can be changed in a further version of MPI >> without backward-compatibility-problems. >> >> I would like to get a reply from >> - the FT group >> - the chapter committee of MPI_Init --> 8. Environmental Manag. >> George Bosilca(c), Josh Hursey, Terry Dontje >> - the chapter committee of MPI_Init_thread --> 12. External Interf. >> Bronis R. de Supinski(c), Pavan Balaji >> >> Before discussing details, I would like to get a clear answer >> whether you believe that this is a good or bad idea. >> >> Best regards >> Rolf >> > > -- > <Mail Attachment.gif> > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle - Performance Technologies > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization
by Josh Hursey 18 Jan '12

18 Jan '12

I like the motivation of the proposal, but I think Terry has a good point. It seems a bit like a hack to repurpose the required/provided arguments to achieve semantic assertions. I would almost prefer some other functionality that must be called before MPI_Init{_thread} that would explicitly set these options. That starts to sound like the assertion ticket that Terry mentioned. So maybe they can be merged or revised. I am also a bit concerned about having conditional semantics in the MPI standard. Though the FT proposal is founded in the condition that the semantics are only meaningful when the error handler is not ARE_FATAL, which is conditional. So I am a bit torn on this point. One thing that your proposal should clearly specify is whether the specified bits must be set to the same value at all processes/threads. Additionally, what if two MPI_COMM_WORLDs connect/accept but have different bits set? Does that restrict how these two world can interact? This was one of the problems posed for the Failhandler_set_mode() semantics that still needs to be addressed here, but in a more general sense. So I think it is an interesting proposal worth considering further. Setting options like 'enable FT' at initialization time (or just before initialization time) might allow the MPI implementation to optimize the library appropriately during setup (choosing different components or algorithms). It might be worth looking at the assertions proposal to see if there is a viable alternative solution there that would achieve the same goals as this proposal without repurposing the required/provided arguments of MPI_Init_thread. -- Josh On Wed, Jan 18, 2012 at 9:32 AM, TERRY DONTJE <terry.dontje(a)oracle.com>wrote: > ** > I think the idea is worthwhile but it really smells similar to the defunct > assertion ticket. I really find piggy-backing the ft, cancel and > any_source modes onto the required/provided bits a little unpleasing to my > senses. The reason I am displeased with the proposal is it seems to > slightly open a door to give an application the ability to give hints and > if we are going to do that we might as well open the door fully and allow > vendor specific hints. Doing the latter will require more than the > require/provided bits. > > The above aside, if the proposal is passed I guess my only other comment > is the of moving the MPI_INIT_THREAD & MPI_QUERY_THREAD to 8.7 (startup) > seems odd to me. I guess I can see the reasoning of moving the interface > to the Startup section but then the threadsafety portion of 12.4.3 section > seems to stick out strangely IMO. > > --td > > > On 1/17/2012 5:31 AM, Rolf Rabenseifner wrote: > > Dear committees of > - FT, > - MPI_Init --> 8. Environmental Management, > - MPI_Init_thread --> 12. External Interfaces. > > Before discussing details, I would like to get a clear answer > whether you believe that the proposal below is a good or bad idea. > > As already mentioned at the Jan. 2012 meeting, I would like to propose > that the FT group may substitute the unclear collective behavior > of MPI_Failhandler_set_mode by adding the mode to the MPI initialization. > > For this, I added a proposal to slide 4 in > MPI_Forum_Overview_MPI-3.0_Jan2012_action-items.ppt > (see my previous mail to the MPI-Forum list) > > - If appropriate, a new ticket that enhances MPI_INIT_THREAD > -- Required and provided as “bit vector” of "bit-wise OR" of > required_/provided_threadsafety > | required_/provided_ft_mode > | required_/provided_cancel_mode > | required_/provided_any_source_mode > -- New mask-constants MPI_THREAD_MASK, MPI_FT_MASK, > MPI_CANCEL_MASK, MPI_ANY_SOURCE_MASK > -- With existing values for required_/provided_threadsafety > MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, > MPI_THREAD_SERIALIZED, and MPI_THREAD_MULTIPLE. > -- With new values for > - required_/provided_ft_mode = > MPI_FT_NONE=0, or > MPI_FT_FAILHANDLER_MODE_ALL≠0, or > MPI_FT_FAILHANDLER_MODE_SUBSET≠0, or > - required_/provided_cancel_mode = > MPI_CANCEL_ALLOWED=0, or > MPI_NO_CANCEL≠0 > - required_/provided_any_source_mode = > MPI_ANY_SOURCE_ALLOWED=0, or > MPI_NO_ANY_SOURCE≠0 > -- Values must be set identical for all processes in an MPI_COMM_WORLD > - It is easier to relax about this in further versions of MPI > than to relax already now and to restrict later > as now done in ticket #222 for required_/provided_threadsafety > - For each of the for "variables", a different decision > can be done. > -- At least for required_/provided_cancel_mode and > ...any_source_mode, I would require that the provided > value must be identical to the required value. > Reason: Internally, the value can be ignored. > -- For required_/provided_ft_mode, I would recommend > to allow that provided_ft_mode must be > - identical to required_ft_mode or MPI_FT_NONE > - and the same in all processes. > -- MPI_INIT_THREAD & MPI_QUERY_THREAD moves > - from 12.4.3 (External Interfaces) > - to 8.7 (Startup) > - but explanations to ...threadsafety | ...ft_mode | > ...cancel_mode | ...any_source_mode are kept or written > in the appropriate sections 12.4.3, new 17.5 (FT Environm.), > 3.8.4 (Cancel), 3.2.4 (Blocking Receive) > -- A call to MPI_INIT is identical to MPI_INIT_THREAD with > - the rules in 12.4.3 about required_/provided_threadsafety > - required_/provided_ft_mode = MPI_FT_NONE, > - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED, > - required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED > -- This ticket would have the following properties: > - It is clearly source-code and ABI backward compatible, > because > - the values of MPI_THREAD_SINGLE, _FUNNELED, ... > need not to be changed and MPI_THREAD_SINGLE need > not to be zero; > - the values representing the current MPI-2.2 quality > are set to zero: MPI_FT_NONE=0, > MPI_CANCEL_ALLOWED=0, and MPI_ANY_SOURCE_ALLOWED=0 > - It is very unlikely that an implementation has used > more than 24 different bits in these 4 integer > constants MPI_THREAD_SINGLE, ... _MULTIPLE. > Therefore MPI_THREAD_MASK would have a maximum > of 24 bits. Enough room for the 4 bits needed > together for the other three ..._MASKs. > - FT can be switched on or off at the MPI initialization > and is switched off in unchanged applications. > Therefore no backward-compatibility-problem with a > modified behavior of the default error handlers > when FT is switched on. > - Normally there should be enough bit-space for further > decisions at MPI initialization. > - The decisions about the cancel and any_source > values would be done in different tickets > - FT quality is optional if we add the > rule that provided_ft_mode may be identical > to required_ft_mode ***or*** MPI_FT_NONE > - This rule can be changed in a further version of MPI > without backward-compatibility-problems. > > I would like to get a reply from > - the FT group > - the chapter committee of MPI_Init --> 8. Environmental Manag. > George Bosilca(c), Josh Hursey, Terry Dontje > - the chapter committee of MPI_Init_thread --> 12. External Interf. > Bronis R. de Supinski(c), Pavan Balaji > > Before discussing details, I would like to get a clear answer > whether you believe that this is a good or bad idea. > > Best regards > Rolf > > > > -- > [image: Oracle] > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle * - Performance Technologies* > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization
by TERRY DONTJE 18 Jan '12

18 Jan '12

I think the idea is worthwhile but it really smells similar to the defunct assertion ticket. I really find piggy-backing the ft, cancel and any_source modes onto the required/provided bits a little unpleasing to my senses. The reason I am displeased with the proposal is it seems to slightly open a door to give an application the ability to give hints and if we are going to do that we might as well open the door fully and allow vendor specific hints. Doing the latter will require more than the require/provided bits. The above aside, if the proposal is passed I guess my only other comment is the of moving the MPI_INIT_THREAD & MPI_QUERY_THREAD to 8.7 (startup) seems odd to me. I guess I can see the reasoning of moving the interface to the Startup section but then the threadsafety portion of 12.4.3 section seems to stick out strangely IMO. --td On 1/17/2012 5:31 AM, Rolf Rabenseifner wrote: > Dear committees of > - FT, > - MPI_Init --> 8. Environmental Management, > - MPI_Init_thread --> 12. External Interfaces. > > Before discussing details, I would like to get a clear answer > whether you believe that the proposal below is a good or bad idea. > > As already mentioned at the Jan. 2012 meeting, I would like to propose > that the FT group may substitute the unclear collective behavior > of MPI_Failhandler_set_mode by adding the mode to the MPI initialization. > > For this, I added a proposal to slide 4 in > MPI_Forum_Overview_MPI-3.0_Jan2012_action-items.ppt > (see my previous mail to the MPI-Forum list) > > - If appropriate, a new ticket that enhances MPI_INIT_THREAD > -- Required and provided as “bit vector” of "bit-wise OR" of > required_/provided_threadsafety > | required_/provided_ft_mode > | required_/provided_cancel_mode > | required_/provided_any_source_mode > -- New mask-constants MPI_THREAD_MASK, MPI_FT_MASK, > MPI_CANCEL_MASK, MPI_ANY_SOURCE_MASK > -- With existing values for required_/provided_threadsafety > MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, > MPI_THREAD_SERIALIZED, and MPI_THREAD_MULTIPLE. > -- With new values for > - required_/provided_ft_mode = > MPI_FT_NONE=0, or > MPI_FT_FAILHANDLER_MODE_ALL≠0, or > MPI_FT_FAILHANDLER_MODE_SUBSET≠0, or > - required_/provided_cancel_mode = > MPI_CANCEL_ALLOWED=0, or > MPI_NO_CANCEL≠0 > - required_/provided_any_source_mode = > MPI_ANY_SOURCE_ALLOWED=0, or > MPI_NO_ANY_SOURCE≠0 > -- Values must be set identical for all processes in an MPI_COMM_WORLD > - It is easier to relax about this in further versions of MPI > than to relax already now and to restrict later > as now done in ticket #222 for required_/provided_threadsafety > - For each of the for "variables", a different decision > can be done. > -- At least for required_/provided_cancel_mode and > ...any_source_mode, I would require that the provided > value must be identical to the required value. > Reason: Internally, the value can be ignored. > -- For required_/provided_ft_mode, I would recommend > to allow that provided_ft_mode must be > - identical to required_ft_mode or MPI_FT_NONE > - and the same in all processes. > -- MPI_INIT_THREAD& MPI_QUERY_THREAD moves > - from 12.4.3 (External Interfaces) > - to 8.7 (Startup) > - but explanations to ...threadsafety | ...ft_mode | > ...cancel_mode | ...any_source_mode are kept or written > in the appropriate sections 12.4.3, new 17.5 (FT Environm.), > 3.8.4 (Cancel), 3.2.4 (Blocking Receive) > -- A call to MPI_INIT is identical to MPI_INIT_THREAD with > - the rules in 12.4.3 about required_/provided_threadsafety > - required_/provided_ft_mode = MPI_FT_NONE, > - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED, > - required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED > -- This ticket would have the following properties: > - It is clearly source-code and ABI backward compatible, > because > - the values of MPI_THREAD_SINGLE, _FUNNELED, ... > need not to be changed and MPI_THREAD_SINGLE need > not to be zero; > - the values representing the current MPI-2.2 quality > are set to zero: MPI_FT_NONE=0, > MPI_CANCEL_ALLOWED=0, and MPI_ANY_SOURCE_ALLOWED=0 > - It is very unlikely that an implementation has used > more than 24 different bits in these 4 integer > constants MPI_THREAD_SINGLE, ... _MULTIPLE. > Therefore MPI_THREAD_MASK would have a maximum > of 24 bits. Enough room for the 4 bits needed > together for the other three ..._MASKs. > - FT can be switched on or off at the MPI initialization > and is switched off in unchanged applications. > Therefore no backward-compatibility-problem with a > modified behavior of the default error handlers > when FT is switched on. > - Normally there should be enough bit-space for further > decisions at MPI initialization. > - The decisions about the cancel and any_source > values would be done in different tickets > - FT quality is optional if we add the > rule that provided_ft_mode may be identical > to required_ft_mode ***or*** MPI_FT_NONE > - This rule can be changed in a further version of MPI > without backward-compatibility-problems. > > I would like to get a reply from > - the FT group > - the chapter committee of MPI_Init --> 8. Environmental Manag. > George Bosilca(c), Josh Hursey, Terry Dontje > - the chapter committee of MPI_Init_thread --> 12. External Interf. > Bronis R. de Supinski(c), Pavan Balaji > > Before discussing details, I would like to get a clear answer > whether you believe that this is a good or bad idea. > > Best regards > Rolf > -- Oracle Terry D. Dontje | Principal Software Engineer Developer Tools Engineering | +1.781.442.2631 Oracle *- Performance Technologies* 95 Network Drive, Burlington, MA 01803 Email terry.dontje(a)oracle.com <mailto:[email protected]>

1 0

Re: [Mpi3-ft] simplified FT proposal
by Sur, Sayantan 17 Jan '12

17 Jan '12

Hi Josh and Rich, Thanks for responding to my email in detail! Since we have a teleconf scheduled tomorrow, we can discuss each point. I want to clarify the motivations behind the rough proposal I made to start the thread. Broadly, my view is that FT is a tough problem and any real solution - especially one that scales - requires work from process launcher/runtime, MPI library, and Applications/other libraries. The manner in which we divide the work between all these components is key. My thoughts are that we should simplify the role of MPI to the extent possible, and have FT-libraries take up some work of providing better semantics to applications. Since there is enough research/debate on-going about ABFT techniques, we might want to simply *enable* fault-tolerant libraries to be built on top of MPI, and only standardize them when there are well-established use cases. The sketch of the model I proposed does not attempt to make every MPI program fault tolerant. Rather, it PERMITS a fault-tolerant version, thereby splitting the work between MPI and its consumers. Let me clarify this with the MPI_ANY_SOURCE master-worker example. "What happens when you comm_check/validate a communicator, post MPI_Recv(ANY) and the rank supposed to send you the message dies? Alternatively, what happens when you have MPI_Recv(ANY) and all other processes have died?" In this model - nothing. The program hangs. The app chose to block wait for any process, and all have died. Tough luck. IMHO, this is a simple behavior. MPI is not trying to be any smarter here than it needs to. However, consider a FT-library which uses a non-blocking receive. The master process keeps a timer of when it last heard from a process. If the timer is larger than expected, try to do a "ping-pong" with the process. If the remote process has indeed failed, then the MPI library returns an error (p2p coordination required from MPI not global). This is a simple example of application-driven heartbeat protocol. Well, the question naturally arises that why make the app/lib do all this work and not do it within MPI? I would argue that if you had to implement the notification-type failure as in the current proposal, it would require you to provide some kind of process-manager backbone that did exactly the same thing in the background. You would need to periodically collect heart-beats from everyone JUST in case someone dies. At scale, you would need to do this constantly and communicate with everyone. The model that I am proposing is actually very similar to yours, it just relaxes the guarantees and semantics. My hope is that the semantics we lose can be provided by not-very-complicated libraries on top of MPI. PS: I agree that MPI_Comm_check() is very similar to the existing MPI_Comm_validate(). I chose a different name as not to confuse the two. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Tuesday, January 17, 2012 12:52 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal Sayantan, Thanks for presentation of an alternative proposal. Alternative proposals are invaluable in helping the group work towards a more refined final proposal for the forum. I do have some questions, comments, and concerns about your proposal detailed below by section. Thanks, Josh (1) Consider a communicator with 1 million processes and 10 die. Would you rather reason about 10 dead or (1M-10) alive? At some point it does not matter much, but for the common case where the number of failures is much less than the size of the MPI universe then it is computationally easier to reason about the set of failed. What is the exact meaning of "is_ok"? (2) The phrase "collectively active/inactive" is bad - as illustrated by the reaction in the meeting. For the discussion consider a MPI_Reduce(). The root may be connected to alive processes in a tree structure, and is waiting patiently for a result. Somewhere in the tree a process fails before entering the MPI_Reduce(). Is it valid for the root to wait forever - locally observed behavior is a hang? or must the processes in the tree propagate the error - locally observed behavior is a transitive error? So considering that situation what must we say in the MPI standard about collective operations? (3) I'm fine with removing most of the failure detector statement if that is what the group wants. It served its purpose as a method to ground the semantics. I feel it is useful to keep in (it is precise and well grounded in theory), but if it has become distracting then we can consider removing it. I'm concerned with the language that "requests may complete with MPI_ERR_PROC_FAIL_STOP" as a way to allude to the option of MPI being able to delay notification. The wording also leaves open the opportunity for an implementation to wait forever to deliver the failure notification thus hanging the program. That is one of the nice aspects of stating explicitly that eventually every process -may- know of the failure, and that communication will complete (as stated in each of the subchapters). But that conversation tends to come back around to well we should make an explicit statement like we have in the document now... (4) Yeah I think Failure Handlers are distracting the conversation, and we should consider bringing them in separately. It would be useful to have a notification of process failure -not- tied to the current MPI call context. Which is what the failure handlers were meant to do. However the following question is what can a user do inside a failure handler, which is what lead to the discussion of the different modes, calling order, ... There is a strong use case for these from UTK to which they can elaborate better than I. (5) So you are proposing the MPI_ANY_SOURCE receives -never- return in error, even if the only process alive in the communicator is the calling process? You suggest that an application should call MPI_Comm_check/validate before calling the MPI_Recv(ANY)? What if a failure occurs between these two operations that would have changed the decision of the application to post the MPI_Recv(ANY) - e.g., the caller is the only process left in the communicator? There remains problems with threading, but much of those problems are derived from the above questions. (6) ok (7) I do not really follow why MPI_Comm_check is better than _validate? Can you elaborate? How does an Allreduce provide you the same semantic guarantees as a MPI_Comm_check/validate? A process might have failed in the tree during the result distribution phase causing dependent processes to fail the Allreduce operation. The point is that MPI_Allreduce is not a fault tolerant agreement algorithm, MPI_Comm_check/validate is. They may have the same failure free complexity, but they are semantically different. (8) The reason we want to allow for the option of reusing a communicator with holes in the membership is for application indexing. This was something that applications requested as many of them use the rank in the communicator to calculate offsets into data structures or participation in the operation. So keeping their rank was seen as important. An application that wants to can create a new communicator with only the alive ranks, but is not required to do so. A different tact, MPI_COMM_WORLD will have holes in it upon process failure, so is it valid to call a collective operation (like MPI_Comm_create) over it to get the new shrunken communicator? (9) Applications may want to keep track of failed processes for application indexing, as just mentioned, but also for later recovery (you need to know who failed if you are going to recover them). (10) I don't think the semantics for MPI_Comm_size are odd at all. MPI_Comm_size returns the total size of the communicator, a value that is often used to setup data structures of appropriate size and to reason about the extent to names in the communicator. So, since communicators have a static size, MPI_Comm_size should always return that static size. If you only track the alive processes, how do you know from which you should recover? (11) So communicator creation should still be 'transactional' - meaning that it returns uniformly at all processes? On Fri, Jan 13, 2012 at 4:41 PM, Sur, Sayantan <sayantan.sur(a)intel.com<mailto:[email protected]>> wrote: FT WG, I would like to thank Josh for enduring the marathon plenary presentation! It was truly commendable. Based on the Forum feedback and vote, it is apparent that there are some significant issues. Primarily due to several new concepts and terms, that the larger Forum does not believe to be required, OR present implementation challenges for the rest of MPI library. I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and resists adding new "features" for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this: for(... really long time ...) { MPI_Comm_check(work_comm, &is_ok, &alive_group); if(!is_ok) { MPI_Comm_create_group(alive_group, ..., &new_comm); // re-balance workload and use new_comm in rest of computation MPI_Comm_free(work_comm); // get rid of old comm work_comm = new_comm; } else { // continue computation using work_comm // if some proc failed in this iteration, roll back work done in this iteration, go back to loop } } Here are some modifications I would like to propose to the current chapter (in order as these concepts/terms appear in the text): 1. Remove concept of "recognized failed" processes. As was pointed out in the meeting, we don't really care about the failed processes, rather the alive ones. Accordingly, rename MPI_Comm(win/file)_validate() to MPI_Comm(win/file)_check(MPI_Comm comm, int * is_ok, MPI_Group * alive_group); 2. Remove concept of "collectively inactive/active". This doesn't really bring anything to the table, rather conflicts with existing definition of collectives. MPI defines collectives as being equivalent of a series of point-to-point calls. As per that definition, if the point-to-point calls succeed (i.e. the corresponding processes are alive), then as locally observed, collective call has also succeeded. As far as the application is concerned as long as the local part of collective is complete successfully, it is OK. If they want to figure out global status, they can always call MPI_Comm_check() or friends. 3. Eventually perfect failure detector/strongly complete/strongly accurate/etc: We replace this discussion (even remove much of 17.3) with a much more straight-forward requirement - "Communication with a process completes with either success or error. In case of communication with failed processes, communication calls and requests may complete with MPI_ERR_PROC_FAILSTOP." Note that MPI standard requires all communication to complete before calling MPI_Finalize - therefore, the first part of this requirement is nothing new. The second part indicates that there is no guarantee that communication with a failed process *will* fail. Messages may have been internally buffered before the real failure may still be delivered per existing MPI semantics. a. This does raise the question from implementers: "When do I mark requests as MPI_ERR_PROC_FAILSTOP? How long do I wait?" The answer completely depends on the implementation. Obviously, there is some requirement to deal with process launcher runtime. In some implementations with connected mode may be able to leverage hw or os techniques to detect connections that have gone down. MPI implementations using connection-less transports may need additional work. However, *none* of this is new work/concepts. As far as possible, we should talk minimally about what the MPI implementation might do to achieve this. 4. Remove process failure handlers - 17.5.1, 17.5.2, 17.5.3, 17.5.4. The only way to find out if something failed is to call MPI_Comm_check() and friends. This removes a whole lot of complexity with failure handlers. Fail handlers can be emulated over this interface as a library. We may consider them for MPI-3.1 (or 4). 5. Point-to-point communication: Remove the concept of MPI_ERR_ANY_SOURCE_DISABLED and corresponding calls to re-enable any source. The concept of disabling ANY_SOURCE is counter-intuitive. When an app/lib posts a recv with ANY_SOURCE, it is specifically telling the MPI library that *any* source is OK and implicitly means that if some senders are unable to send, application/lib does not care! Master/slave type of applications wishing to use FT features can periodically call MPI_Comm_check(). Additionally, if the master tries to send to the dead process, it may get an error. My guess is that master/slave type of apps are among the most resilient, and some even work with the current standard (MPI_ERRORS_RETURN). A benefit of removing this restriction is that we no longer have the threading complexities of re-enabling any source using reader/writer locks :) Therefore, we can remove 17.6.3. 6. Retain MPI_Comm_drain() and MPI_Comm_idrain() as they provide useful functionality. 7. Collective communication: Rename comm_validate() to comm_check() as per discussion above. We can keep comm_check_multiple() as it provides useful functionality for overlapping communicators by reducing overhead to check them. We can retain much of 17.7.2 while removing references to "collectively inactive". If the output of collective depends on contribution from a failed process, then obviously, the collective fails. This is in keeping with point-to-point semantics - one cannot receive any data from a failed process. Keep in mind if the contribution from failed process may have arrived before it failed - and that is OK (not flagged as failure). Some collectives, such as MPI_Bcast, may succeed even if processes down the bcast tree have failed as sends may simply be buffered. The app/lib will only know if a collective was a global success by either performing an Allreduce after the collective OR calling comm_check(). In any case, it is left to app/lib and not MPI to report failures of processes the library didn't try to communicate with during this op. 8. I am proposing that once a collective fails with MPI_ERR_PROC_FAIL_STOP, all subsequent collectives on that comm fail immediately with MPI_ERR_PROC_FAIL_STOP. App/lib needs to use MPI_Comm_create_group() to fork off a new comm of live procs and continue with it. This is a deviation from the current proposal that allows collectives on bad comms (after re-enabling collectives) and keeps 0s as contributions. I am aware that this might not fully satisfy all use cases (although at this point of time, I cannot think of any), but in a broader view, we could think this of as a compromise to reduce complexity. 9. Example 17.4 changes only slightly to call comm_check() and then split off the new communicator. Why keep failed procs in the communicator anyways? 10. Note that this change in semantics allows us to bypass the question raised: "Why does comm_size() on a communicator with failed procs still return the old value- alive_ranks + failed_ranks?" As I mentioned before, this is odd, and we should encourage app/lib to only deal with known alive ranks. The current proposal does the reverse - forces app to keep track of "known failed". This causes confusion! 11. Process topologies 17.9 - should change to say that we can only use communicators with live ranks. i.e. if you know your comm was bad, split off a new comm with live ranks. During the op, some ranks may fail - and that is OK since MPI_ERR_PROC_FAIL_STOP will be raised. This is mentioned in the current proposal. 12. Similar changes in semantics to windows and files. Please let me know if I overlooked some corner cases or I have mis-interpreted the text of the current chapter. I gave it some thought, but WG knows best! Thanks! === Sayantan Sur, Ph.D. Intel Corp. _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] simplified FT proposal
by George R Carr Jr 17 Jan '12

17 Jan '12

Hello, I'm not sure that this really covers the use case I presented at the July, 2011 meeting. Mine is a fault tolerant case (not fault recovery) for a semi-real-timeish processing situation. The application must conclude within a max wall clock limit. Think of this as having multiple processing components, not all of which need complete for the result to be useful. Testing at the beginning of an operation is not sufficient. The kind of process I have implemented (not in MPI) is: for (... really long time ... ) { communicate-nowait if no ack/completion in set time then declare failure of that other component{ test communicator(s) affected, checking for component failures if (continuation makes sense){ rebuild affected communicator(s) else exit } } } The timeout feature is critical. Regards, George >Content-Language: en-US >Content-Type: multipart/alternative; > > boundary="_000_CE01B5B59D416C4BA3228140336D1DE401367BORSMSX103amrcorpi_" > >FT WG, > >I would like to thank Josh for enduring the >marathon plenary presentation! It was truly >commendable. > >Based on the Forum feedback and vote, it is >apparent that there are some significant issues. >Primarily due to several new concepts and terms, >that the larger Forum does not believe to be >required, OR present implementation challenges >for the rest of MPI library. > >I would like to argue for a simplified version >of the proposal that covers a large percentage >of use-cases and resists adding new "features" >for the full-range of ABFT techniques. It is >good if we have a more pragmatic view and not >sacrifice the entire FT proposal for the 1% >fringe cases. Most apps just want to do >something like this: > >for( really long time ) { > MPI_Comm_check(work_comm, &is_ok, &alive_group); > if(!is_ok) { > MPI_Comm_create_group(alive_group, , &new_comm); > // re-balance workload and use new_comm in rest of computation > MPI_Comm_free(work_comm); // get rid of old comm > work_comm = new_comm; > } else { > // continue computation using work_comm > // if some proc failed in this iteration, >roll back work done in this iteration, go back >to loop > } >} > >Here are some modifications I would like to >propose to the current chapter (in order as >these concepts/terms appear in the text): > >1. Remove concept of "recognized failed" >processes. As was pointed out in the meeting, we >don't really care about the failed processes, >rather the alive ones. Accordingly, rename >MPI_Comm(win/file)_validate() to >MPI_Comm(win/file)_check(MPI_Comm comm, int * >is_ok, MPI_Group * alive_group); >2. Remove concept of "collectively >inactive/active". This doesn't really bring >anything to the table, rather conflicts with >existing definition of collectives. MPI defines >collectives as being equivalent of a series of >point-to-point calls. As per that definition, if >the point-to-point calls succeed (i.e. the >corresponding processes are alive), then as >locally observed, collective call has also >succeeded. As far as the application is >concerned as long as the local part of >collective is complete successfully, it is OK. >If they want to figure out global status, they >can always call MPI_Comm_check() or friends. >3. Eventually perfect failure >detector/strongly complete/strongly >accurate/etc: We replace this discussion (even >remove much of 17.3) with a much more >straight-forward requirement - "Communication >with a process completes with either success or >error. In case of communication with failed >processes, communication calls and requests may >complete with MPI_ERR_PROC_FAILSTOP." Note that >MPI standard requires all communication to >complete before calling MPI_Finalize - >therefore, the first part of this requirement is >nothing new. The second part indicates that >there is no guarantee that communication with a >failed process *will* fail. Messages may have >been internally buffered before the real failure >may still be delivered per existing MPI >semantics. >a. This does raise the question from >implementers: "When do I mark requests as >MPI_ERR_PROC_FAILSTOP? How long do I wait?" The >answer completely depends on the implementation. >Obviously, there is some requirement to deal >with process launcher runtime. In some >implementations with connected mode may be able >to leverage hw or os techniques to detect >connections that have gone down. MPI >implementations using connection-less transports >may need additional work. However, *none* of >this is new work/concepts. As far as possible, >we should talk minimally about what the MPI >implementation might do to achieve this. >4. Remove process failure handlers - >17.5.1, 17.5.2, 17.5.3, 17.5.4. The only way to >find out if something failed is to call >MPI_Comm_check() and friends. This removes a >whole lot of complexity with failure handlers. >Fail handlers can be emulated over this >interface as a library. We may consider them for >MPI-3.1 (or 4). >5. Point-to-point communication: Remove >the concept of MPI_ERR_ANY_SOURCE_DISABLED and >corresponding calls to re-enable any source. The >concept of disabling ANY_SOURCE is >counter-intuitive. When an app/lib posts a recv >with ANY_SOURCE, it is specifically telling the >MPI library that *any* source is OK and >implicitly means that if some senders are unable >to send, application/lib does not care! >Master/slave type of applications wishing to use >FT features can periodically call >MPI_Comm_check(). Additionally, if the master >tries to send to the dead process, it may get an >error. My guess is that master/slave type of >apps are among the most resilient, and some even >work with the current standard >(MPI_ERRORS_RETURN). A benefit of removing this >restriction is that we no longer have the >threading complexities of re-enabling any source >using reader/writer locks J Therefore, we can >remove 17.6.3. >6. Retain MPI_Comm_drain() and >MPI_Comm_idrain() as they provide useful >functionality. >7. Collective communication: Rename >comm_validate() to comm_check() as per >discussion above. We can keep >comm_check_multiple() as it provides useful >functionality for overlapping communicators by >reducing overhead to check them. We can retain >much of 17.7.2 while removing references to >"collectively inactive". If the output of >collective depends on contribution from a failed >process, then obviously, the collective fails. >This is in keeping with point-to-point semantics >- one cannot receive any data from a failed >process. Keep in mind if the contribution from >failed process may have arrived before it failed >- and that is OK (not flagged as failure). Some >collectives, such as MPI_Bcast, may succeed even >if processes down the bcast tree have failed as >sends may simply be buffered. The app/lib will >only know if a collective was a global success >by either performing an Allreduce after the >collective OR calling comm_check(). In any case, >it is left to app/lib and not MPI to report >failures of processes the library didn't try to >communicate with during this op. >8. I am proposing that once a collective >fails with MPI_ERR_PROC_FAIL_STOP, all >subsequent collectives on that comm fail >immediately with MPI_ERR_PROC_FAIL_STOP. App/lib >needs to use MPI_Comm_create_group() to fork off >a new comm of live procs and continue with it. >This is a deviation from the current proposal >that allows collectives on bad comms (after >re-enabling collectives) and keeps 0s as >contributions. I am aware that this might not >fully satisfy all use cases (although at this >point of time, I cannot think of any), but in a >broader view, we could think this of as a >compromise to reduce complexity. >9. Example 17.4 changes only slightly to >call comm_check() and then split off the new >communicator. Why keep failed procs in the >communicator anyways? >10. Note that this change in semantics allows >us to bypass the question raised: "Why does >comm_size() on a communicator with failed procs >still return the old value- alive_ranks + >failed_ranks?" As I mentioned before, this is >odd, and we should encourage app/lib to only >deal with known alive ranks. The current >proposal does the reverse - forces app to keep >track of "known failed". This causes confusion! >11. Process topologies 17.9 - should change to >say that we can only use communicators with live >ranks. i.e. if you know your comm was bad, split >off a new comm with live ranks. During the op, >some ranks may fail - and that is OK since >MPI_ERR_PROC_FAIL_STOP will be raised. This is >mentioned in the current proposal. >12. Similar changes in semantics to windows and files. > >Please let me know if I overlooked some corner >cases or I have mis-interpreted the text of the >current chapter. I gave it some thought, but WG >knows best! > > >Thanks! > >=== >Sayantan Sur, Ph.D. >Intel Corp. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- George R Carr Jr U.S. Department of Commerce NOAA/OAR/GSD, DSRC R/GSD2, 2B142 Tel (best): 303-325-3334 Tel (NOAA): 303-497-4714 Email: George.Carr(a)noaa.gov

1 0

Re: [Mpi3-ft] simplified FT proposal
by Josh Hursey 17 Jan '12

17 Jan '12

The call-in information is posted the day before every teleconf to the mailing list. Our next scheduled meeting if Feb 1, but I suspect we will choose to meet next week (Jan. 25) given the amount of things we have to discuss. Though that will be decided in tomorrows teleconf. -- Josh On Tue, Jan 17, 2012 at 4:44 PM, Anthony Skjellum <tony(a)cis.uab.edu> wrote: > Josh, I can't come to tomorrow's, but I will attend next week's, January > 25, ok? > Where is call-in info available? I will have a basic document by Jan 25. > > Thanks, > Tony > > > ----- Original Message ----- > From: "Josh Hursey" <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" < > mpi3-ft(a)lists.mpi-forum.org> > Sent: Tuesday, January 17, 2012 3:35:06 PM > Subject: Re: [Mpi3-ft] simplified FT proposal > > > The next teleconf is tomorrow at noon. I'm sendin gout the announcement > shortly. > > > -- Josh > > > On Tue, Jan 17, 2012 at 4:21 PM, Anthony Skjellum < tony(a)cis.uab.edu > > wrote: > > > Josh, I just sent you a note, agreeing to do a writeup. I need a few days. > When is next telecon? > > Tony > > > > > ----- Original Message ----- > From: "Josh Hursey" < jjhursey(a)open-mpi.org > > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" < > mpi3-ft(a)lists.mpi-forum.org > > Sent: Tuesday, January 17, 2012 3:15:13 PM > Subject: Re: [Mpi3-ft] simplified FT proposal > > > Tony, > > > I have to admit that I am getting a bit lost in your multipart > presentation of your proposal. So that is making it difficult for me to > reason about it as a whole. Can you synthesis your proposal (either as a > separate email thread or on the wiki) and maybe we can discuss it further > on the teleconf and mailing list? That will give us a single place to point > to and refine during the subsequent discussions. > > > Thanks, > Josh > > > On Mon, Jan 16, 2012 at 4:18 PM, Anthony Skjellum < > tony(a)runtimecomputing.com > wrote: > > > > More > > > #5) > > > A parallel thread, running on the same group as the communicator used for > the user program in FTBLOCK, would be free to be running a gossip protocol, > instantiated > a) by the user at his/her choice > b) by MPI at its choice, by user demand > > > This parallel inspector thread propagates error state, and accepts input > to the error state by allowing user programs to assert error state within > the FTBLOCK. > > > You can think of this parallel thread as cooperation of the progress > engines with gossip, or as a completely optional activity, created by the > user, or as > a part of the FT proposal. As long as there are local ways to add error > state, we can write this as a layered library for the sake of making sure > that local > errors are propagated (remembering that MPI is returning local errors to > the user inside his/her FTBLOCK). > > > If we want the implementation to provide/support this kind of FT-friendly > error propagation, that is a bigger step, which I am not advocating as > required. > I think it would be nice to have this be part of a fault tolerant > communication space. But, I am OK with users doing this for themselves, > esp. because they > can control granularity better. > > > #6) I think MPI_COMM_WORLD is a bummer in the FT world. We don't want it > to hang around very long after we get going. If we are really working on > subgroups, and these subgroups form a hierarchical graph, rather than an > all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD > after > the first process breaks. So - as an example of pain of MPI scalability - > the build-down model of MPI-1 is less good for this than a build-up > model, where we find a way for groups to rendezvous. Obviously, for > convenience, the all-to-all virtual topology in the onset of MPI_Init() is > nice, > but I am assuming that errors may happen quite quickly. > > > MPI_Init() with no failures is only good for a limited window of time, > given failure rates. During this time, we would like to "turn off > MPI_COMM_WORLD" > unless we really don't need it, or at least not have to reconstruct it > ever if we don't need it. > > > So, we need to agree on a fault-friendly MPI_Init()... > > > An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of > spawning MPI, or the spawn command that creates the world, create an > effective communicator. We should generalize MPI_Init() to support outcomes > other than pure success, such as partial success (e.g., smaller world than > stipulated). > > > A really good approach would be to only create the actual communicators > you really need to run your process, not the virtual-all-to-all world of > MPI-1 and MPI-2. > But, that is a heresy, so forgive me. > > > Tony > > > > > > > > > > > > > > > > On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum < > tony(a)runtimecomputing.com > wrote: > > > Satantan, > > > Here are my further thoughts, which hopefully includes an answer to your > questions :-) > > > 1) Everything is set to "ERRORS return"; the goal is to get local errors > back if they are available. > > > 2) I would emphasize non-blocking operations, but blocking operations > implemented with internal timeout could return a timeout-type error > > > 3) You don't have to return the same return code, or the same results in > all processes in the communicator, you can get erroneous results > or local failures; the functions are also allowed to produce incorrect > results [and we should then discuss what error reporting means here... > I am happy with local errors returned where known, recognizing those > processes may die before the bottom of the block. However, I also > expect the implementation to do its best to propagate error state > knowledge within this FTBLOCK, based organically on ongoing communication > or on gossip if an implementation so chose.] > > > Also, because we assume that there is also algorithmic fault tolerance at > work, local errors may be raised by the application because it is doing > checks > for validity etc. > > > So, either MPI or the application may raise local errors prior to the > bottom of the FTBLOCK, and the success of the bottom of the block must be > allowed > to be fail, just based on ABFT inputs from the application to MPI, not > just based on MPI's opinion. > > > 4) If you are willing to do everything nonblocking, then I can describe > the test at the bottom of the FTBLOCK as follows: > > > The test operation at the bottom of the FTBLOCK is effectively a > generalized WAIT_ALL, that completes or doesn't complete all the outstanding > requests, returning errors related to the faults observed, and providing a > unified 0/1 success failure state consistently across the group of comm > [or surviving members thereof]. > > > In my view, the application as well as MPI can contribute error state as > input to the FTBLOCK test. > > > Also, the application that gets local errors inside the loop already is > immediately ready to punt at that point, and do a jump to the bottom of the > loop. Let's > assume it is required to do all the operations before getting to the > bottom of the loop, for now, and we just allow that some of these may > return further > errors (I am trying to keep it simple), and MPI-like rules of all > attempting the operations. If we can get this to work, we can weaken it > later. > > > There is no good way to describe some BLOCKING, some nonblocking > operations, because we have no descriptor to tell us if something failed > that previously > returned... and did not give a local error, so I am not going to pursue > BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this later, > if we can get a consistent solution using all nonblocking. > > > Please tell me what you think. > > > Thanks for responding! > > > Tony > > > > > > > On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan < sayantan.sur(a)intel.com> wrote: > > > > > > > Hi Tony, > > > > In the example semantics you mentioned, are the “ops” required to return > the same result on all processors? Although this doesn’t change the API > “op”, but it does change completion semantics of almost all MPI ops. I hope > I am correctly interpreting your message. > > > > > Thanks. > > > > === > > Sayantan Sur, Ph.D. > > Intel Corp. > > > > > > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of Anthony Skjellum > Sent: Sunday, January 15, 2012 7:06 PM > > > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] simplified FT proposal > > > > > > > > Everyone, I think we need to start from scratch. > > > > > > We should look for minimal fault-tolerant models that are achievable and > useful. They may allow for a combination of faults (process and network), > but in the end, as discussed in San Jose: > > > > > > FTBLOCK > > > -------------- > > > Start_Block(comm) > > > > > > op [normal MPI operation on communicator specified by Start_Block, or > subset thereof] > > > op > > > op > > > > > > Test_Block(comm) > > > > > > Which either succeeds, or fails, on the whole list of operations, followed > by ways to reconstruct communicators and add back processes (easily), > > > provides for 3-level fault tolerant model > > > > > > a) Simply retry if the kind of error at the Test_Block is retryable > > > b) Simply reconstruct the communicator, use algorithmic fault tolerance to > get lost data, and retry the block > > > c) drop back to 1 or more levels of checkpoint-restart. > > > > > > We can envision in this model an unrolling of work, in terms of a > parameter N, if there is a lot of vector work, to allow granularity control > > > as a function of fault environment. > > > > > > In some sense, a simpler model, that provides for efforts to detect > > > a) by MPI > > > b) allowing application monitor to assert failure asynchronously to this > loop > > > > > > provides a more general ability to have coverage of faults, including but > not limited to process faults and possible network faults. > > > > > > It changes the API very little. > > > > > > It motivates the use of buffers, not zero copy, to support the fact that > you may to roll-back a series of operations, thereby revealing fault-free > overhead directly. > > > > > > Start_Block and Test_Block are collective and synchronizing, such as > barriers. Because we have uncertainty to within a message, multiple > barriers (as mentioned by George Bosilca K to me in a sidebar at the > meeting). > > > > > > We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we > can. Once we have this stable intermediate form, we explore more > > > options. > > > > > > I think it is important to recognize that the reconstruction step, > including readding processes and making new communicators may mean smarter > Join operations. It is clear we need to be able to treat failures during > the recovery process, and use a second level loop, possibly bombing out > > > to checkpoint, if we cannot make net progress on recovery because of > unmodeled error issues. > > > > > > The testing part leverages all the learning so far, but needn't be > restricted to modeled errors like process faults. There can be modeled and > unmodeled faults. Based on what fault comes up, the user application then > has to decide how hard a retry to do, whether just to add processes, > > > whether just to retry the loop, whether to go to a checkpoint, whether to > restart the APP. MPI could give advice, based on its understanding of > > > the fault model, in terms of sufficient conditions for "working harder" vs > "trying the easiest" for fault models it understands somewhat for a system. > > > > > > Now, the comments here are a synopsis of part of the side bars and open > discussion we had in San Jose, distilled a bit. I want to know why > > > we can't start with this, succeed with this, implement and test it, and > having succeeded, do more in a future 3.y, y > x release, given user > experience. > > > > > > I am not speaking to the choice of "killing all communicators" as with > FT-MPI, or "just remaking those you need to remake." I think we need to > resolve. Honestly, groups own the fault property, not communicators, and > all groups held by communicators where the fault happened should be > rebuilt, not all communicators... Let's argue on that. > > > > > > So, my suggestion is REBOOT the proposal with something along lines above, > unless you all see this is no better. > > > > > > With kind regards, > > > Tony > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan < sayantan.sur(a)intel.com > > wrote: > > > > Hi Bill, > > > > I am in agreement with your suggestion to have a collective over a > communicator that returns a new communicator containing ranks “alive some > point during construction”. It provides cleaner semantics. The example was > merely trying to utilize the new MPI_Comm_create_group API that the Forum > is considering. > > > > MPI_Comm_check provides a method to form global consensus in that all > ranks in comm did call it. It does not imply anything about current status > of comm, or even the status “just before” the call returns. During the > interval before the next call to MPI_Comm_check, it is possible that many > ranks fail. However, the app/lib using MPI knows the point where everyone > was alive. > > > > Thanks. > > > > > > === > > Sayantan Sur, Ph.D. > > Intel Corp. > > > > > > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of William Gropp > Sent: Sunday, January 15, 2012 2:41 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] simplified FT proposal > > > > > > One concern that I have with fault tolerant proposals has to do with races > in the specification. This is an area where users often "just want it to > work" but getting it right is tricky. In the example here, the > "alive_group" is really only that at some moment shortly before > "MPI_Comm_check" returns (and possibly not even that). After that, it is > really the "group_of_processes_that_was_alive_at_some_point_in_the_past". > Since there are sometimes correlations in failures, this could happen even > if the initial failure is rare. An alternate form might be to have a > routine, collective over a communicator, that returns a new communicator > meeting some definition of "members were alive at some point during > construction". It wouldn't guarantee you could use it, but it would have > cleaner semantics. > > > > > > > Bill > > > > > > > On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote: > > > > > I would like to argue for a simplified version of the proposal that covers > a large percentage of use-cases and resists adding new “features” for the > full-range of ABFT techniques. It is good if we have a more pragmatic view > and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps > just want to do something like this: > > > > > > for(… really long time …) { > > > MPI_Comm_check(work_comm, &is_ok, &alive_group); > > > if(!is_ok) { > > > MPI_Comm_create_group(alive_group, …, &new_comm); > > > // re-balance workload and use new_comm in rest of computation > > > MPI_Comm_free(work_comm); // get rid of old comm > > > work_comm = new_comm; > > > } else { > > > // continue computation using work_comm > > > // if some proc failed in this iteration, roll back work done in this > iteration, go back to loop > > > } > > > } > > > > > > > > > > William Gropp > > > Director, Parallel Computing Institute > > > Deputy Director for Research > > > Institute for Advanced Computing Applications and Technologies > > > Paul and Cynthia Saylor Professor of Computer Science > > > University of Illinois Urbana-Champaign > > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > > > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > > > > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > Anthony Skjellum, PhD > Professor and Chair > Dept. of Computer and Information Sciences > Director, UAB Center for Information Assurance and Joint Forensic Research > ("The Center") > University of Alabama at Birmingham > +1-(205)934-8657 ; FAX: +1- (205)934-5473 > > ___________________________________________ > CONFIDENTIALITY: This e-mail and any attachments are confidential and > may be privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person, > use it for any purpose or store or copy the information in any medium. > > Please consider the environment before printing this e-mail > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > Anthony Skjellum, PhD > Professor and Chair > Dept. of Computer and Information Sciences > Director, UAB Center for Information Assurance and Joint Forensic Research > ("The Center") > University of Alabama at Birmingham > +1-(205)934-8657; FAX: +1- (205)934-5473 > > ___________________________________________ > CONFIDENTIALITY: This e-mail and any attachments are confidential and > may be privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person, > use it for any purpose or store or copy the information in any medium. > > Please consider the environment before printing this e-mail > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] simplified FT proposal
by Anthony Skjellum 17 Jan '12

17 Jan '12

Josh, I can't come to tomorrow's, but I will attend next week's, January 25, ok? Where is call-in info available? I will have a basic document by Jan 25. Thanks, Tony ----- Original Message ----- From: "Josh Hursey" <jjhursey(a)open-mpi.org> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Sent: Tuesday, January 17, 2012 3:35:06 PM Subject: Re: [Mpi3-ft] simplified FT proposal The next teleconf is tomorrow at noon. I'm sendin gout the announcement shortly. -- Josh On Tue, Jan 17, 2012 at 4:21 PM, Anthony Skjellum < tony(a)cis.uab.edu > wrote: Josh, I just sent you a note, agreeing to do a writeup. I need a few days. When is next telecon? Tony ----- Original Message ----- From: "Josh Hursey" < jjhursey(a)open-mpi.org > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" < mpi3-ft(a)lists.mpi-forum.org > Sent: Tuesday, January 17, 2012 3:15:13 PM Subject: Re: [Mpi3-ft] simplified FT proposal Tony, I have to admit that I am getting a bit lost in your multipart presentation of your proposal. So that is making it difficult for me to reason about it as a whole. Can you synthesis your proposal (either as a separate email thread or on the wiki) and maybe we can discuss it further on the teleconf and mailing list? That will give us a single place to point to and refine during the subsequent discussions. Thanks, Josh On Mon, Jan 16, 2012 at 4:18 PM, Anthony Skjellum < tony(a)runtimecomputing.com > wrote: More #5) A parallel thread, running on the same group as the communicator used for the user program in FTBLOCK, would be free to be running a gossip protocol, instantiated a) by the user at his/her choice b) by MPI at its choice, by user demand This parallel inspector thread propagates error state, and accepts input to the error state by allowing user programs to assert error state within the FTBLOCK. You can think of this parallel thread as cooperation of the progress engines with gossip, or as a completely optional activity, created by the user, or as a part of the FT proposal. As long as there are local ways to add error state, we can write this as a layered library for the sake of making sure that local errors are propagated (remembering that MPI is returning local errors to the user inside his/her FTBLOCK). If we want the implementation to provide/support this kind of FT-friendly error propagation, that is a bigger step, which I am not advocating as required. I think it would be nice to have this be part of a fault tolerant communication space. But, I am OK with users doing this for themselves, esp. because they can control granularity better. #6) I think MPI_COMM_WORLD is a bummer in the FT world. We don't want it to hang around very long after we get going. If we are really working on subgroups, and these subgroups form a hierarchical graph, rather than an all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD after the first process breaks. So - as an example of pain of MPI scalability - the build-down model of MPI-1 is less good for this than a build-up model, where we find a way for groups to rendezvous. Obviously, for convenience, the all-to-all virtual topology in the onset of MPI_Init() is nice, but I am assuming that errors may happen quite quickly. MPI_Init() with no failures is only good for a limited window of time, given failure rates. During this time, we would like to "turn off MPI_COMM_WORLD" unless we really don't need it, or at least not have to reconstruct it ever if we don't need it. So, we need to agree on a fault-friendly MPI_Init()... An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of spawning MPI, or the spawn command that creates the world, create an effective communicator. We should generalize MPI_Init() to support outcomes other than pure success, such as partial success (e.g., smaller world than stipulated). A really good approach would be to only create the actual communicators you really need to run your process, not the virtual-all-to-all world of MPI-1 and MPI-2. But, that is a heresy, so forgive me. Tony On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum < tony(a)runtimecomputing.com > wrote: Satantan, Here are my further thoughts, which hopefully includes an answer to your questions :-) 1) Everything is set to "ERRORS return"; the goal is to get local errors back if they are available. 2) I would emphasize non-blocking operations, but blocking operations implemented with internal timeout could return a timeout-type error 3) You don't have to return the same return code, or the same results in all processes in the communicator, you can get erroneous results or local failures; the functions are also allowed to produce incorrect results [and we should then discuss what error reporting means here... I am happy with local errors returned where known, recognizing those processes may die before the bottom of the block. However, I also expect the implementation to do its best to propagate error state knowledge within this FTBLOCK, based organically on ongoing communication or on gossip if an implementation so chose.] Also, because we assume that there is also algorithmic fault tolerance at work, local errors may be raised by the application because it is doing checks for validity etc. So, either MPI or the application may raise local errors prior to the bottom of the FTBLOCK, and the success of the bottom of the block must be allowed to be fail, just based on ABFT inputs from the application to MPI, not just based on MPI's opinion. 4) If you are willing to do everything nonblocking, then I can describe the test at the bottom of the FTBLOCK as follows: The test operation at the bottom of the FTBLOCK is effectively a generalized WAIT_ALL, that completes or doesn't complete all the outstanding requests, returning errors related to the faults observed, and providing a unified 0/1 success failure state consistently across the group of comm [or surviving members thereof]. In my view, the application as well as MPI can contribute error state as input to the FTBLOCK test. Also, the application that gets local errors inside the loop already is immediately ready to punt at that point, and do a jump to the bottom of the loop. Let's assume it is required to do all the operations before getting to the bottom of the loop, for now, and we just allow that some of these may return further errors (I am trying to keep it simple), and MPI-like rules of all attempting the operations. If we can get this to work, we can weaken it later. There is no good way to describe some BLOCKING, some nonblocking operations, because we have no descriptor to tell us if something failed that previously returned... and did not give a local error, so I am not going to pursue BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this later, if we can get a consistent solution using all nonblocking. Please tell me what you think. Thanks for responding! Tony On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan < sayantan.sur(a)intel.com > wrote: Hi Tony, In the example semantics you mentioned, are the “ops” required to return the same result on all processors? Although this doesn’t change the API “op”, but it does change completion semantics of almost all MPI ops. I hope I am correctly interpreting your message. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of Anthony Skjellum Sent: Sunday, January 15, 2012 7:06 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal Everyone, I think we need to start from scratch. We should look for minimal fault-tolerant models that are achievable and useful. They may allow for a combination of faults (process and network), but in the end, as discussed in San Jose: FTBLOCK -------------- Start_Block(comm) op [normal MPI operation on communicator specified by Start_Block, or subset thereof] op op Test_Block(comm) Which either succeeds, or fails, on the whole list of operations, followed by ways to reconstruct communicators and add back processes (easily), provides for 3-level fault tolerant model a) Simply retry if the kind of error at the Test_Block is retryable b) Simply reconstruct the communicator, use algorithmic fault tolerance to get lost data, and retry the block c) drop back to 1 or more levels of checkpoint-restart. We can envision in this model an unrolling of work, in terms of a parameter N, if there is a lot of vector work, to allow granularity control as a function of fault environment. In some sense, a simpler model, that provides for efforts to detect a) by MPI b) allowing application monitor to assert failure asynchronously to this loop provides a more general ability to have coverage of faults, including but not limited to process faults and possible network faults. It changes the API very little. It motivates the use of buffers, not zero copy, to support the fact that you may to roll-back a series of operations, thereby revealing fault-free overhead directly. Start_Block and Test_Block are collective and synchronizing, such as barriers. Because we have uncertainty to within a message, multiple barriers (as mentioned by George Bosilca K to me in a sidebar at the meeting). We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we can. Once we have this stable intermediate form, we explore more options. I think it is important to recognize that the reconstruction step, including readding processes and making new communicators may mean smarter Join operations. It is clear we need to be able to treat failures during the recovery process, and use a second level loop, possibly bombing out to checkpoint, if we cannot make net progress on recovery because of unmodeled error issues. The testing part leverages all the learning so far, but needn't be restricted to modeled errors like process faults. There can be modeled and unmodeled faults. Based on what fault comes up, the user application then has to decide how hard a retry to do, whether just to add processes, whether just to retry the loop, whether to go to a checkpoint, whether to restart the APP. MPI could give advice, based on its understanding of the fault model, in terms of sufficient conditions for "working harder" vs "trying the easiest" for fault models it understands somewhat for a system. Now, the comments here are a synopsis of part of the side bars and open discussion we had in San Jose, distilled a bit. I want to know why we can't start with this, succeed with this, implement and test it, and having succeeded, do more in a future 3.y, y > x release, given user experience. I am not speaking to the choice of "killing all communicators" as with FT-MPI, or "just remaking those you need to remake." I think we need to resolve. Honestly, groups own the fault property, not communicators, and all groups held by communicators where the fault happened should be rebuilt, not all communicators... Let's argue on that. So, my suggestion is REBOOT the proposal with something along lines above, unless you all see this is no better. With kind regards, Tony On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan < sayantan.sur(a)intel.com > wrote: Hi Bill, I am in agreement with your suggestion to have a collective over a communicator that returns a new communicator containing ranks “alive some point during construction”. It provides cleaner semantics. The example was merely trying to utilize the new MPI_Comm_create_group API that the Forum is considering. MPI_Comm_check provides a method to form global consensus in that all ranks in comm did call it. It does not imply anything about current status of comm, or even the status “just before” the call returns. During the interval before the next call to MPI_Comm_check, it is possible that many ranks fail. However, the app/lib using MPI knows the point where everyone was alive. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of William Gropp Sent: Sunday, January 15, 2012 2:41 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal One concern that I have with fault tolerant proposals has to do with races in the specification. This is an area where users often "just want it to work" but getting it right is tricky. In the example here, the "alive_group" is really only that at some moment shortly before "MPI_Comm_check" returns (and possibly not even that). After that, it is really the "group_of_processes_that_was_alive_at_some_point_in_the_past". Since there are sometimes correlations in failures, this could happen even if the initial failure is rare. An alternate form might be to have a routine, collective over a communicator, that returns a new communicator meeting some definition of "members were alive at some point during construction". It wouldn't guarantee you could use it, but it would have cleaner semantics. Bill On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote: I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and resists adding new “features” for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this: for(… really long time …) { MPI_Comm_check(work_comm, &is_ok, &alive_group); if(!is_ok) { MPI_Comm_create_group(alive_group, …, &new_comm); // re-balance workload and use new_comm in rest of computation MPI_Comm_free(work_comm); // get rid of old comm work_comm = new_comm; } else { // continue computation using work_comm // if some proc failed in this iteration, roll back work done in this iteration, go back to loop } } William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensic Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657 ; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensic Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 17 Jan '12

17 Jan '12

Reminder that we have a teleconf tomorrow, Wednesday 1/18/2012, at Noon EST. Date: January 18, 2012 Time: Noon EST/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - MPI Forum recap and discussion - Open discussion from email threads on the mailing list - Other items? Additional Agenda Items [as time permits]: - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Piggybacking? (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Jan. 18, 2012 (Teleconf) * Feb. 1, 2012 (Teleconf) * Feb. 15, 2012 (Teleconf) * Feb. 29, 2012 (Teleconf) * March 5, 2012 (MPI Forum Meeting) * March 14, 2012 (Teleconf) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0