January 2012 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Matching MPI communication object creation across process failure
by Josh Hursey 25 Jan '12

25 Jan '12

The current proposal states that MPI object creation functions (e.g., MPI_Comm_create, MPI_Win_create, MPI_File_open): ------------------- All participating communicator(s) must be collectively active before calling any communicator creation operation. Otherwise, the communicator creation operation will uniformly raise an error code of the class MPI_ERR_PROC_FAIL_STOP. If a process failure prevents the uniform creation of the communicator then the communicator construction operation must ensure that the communicator is not created, and all alive participating processes will raise an error code of the class MPI_ERR_PROC_FAIL_STOP. Communicator construction operations will match across the notification of a process failure. As such, all alive processes must call the communicator construction operations the same number of times regardless of whether the emergent process failure makes the call irrelevant to the application. ------------------- So there are three points here: (1) That the communicator must be 'collectively active' before calling the operation, (2) Uniform creation of the communication object, and (3) Creation operations match across process failure. Point (2) seems to be necessary so that all processes only ever see a communication object that is consistent across all processes. This implies a fault tolerant agreement protocol (group membership). There was a question about why point (3) is necessary. We (Darius, UTK, and I) discussed this on 12/12/2011, and I posted my notes on this to the list: http://lists.mpi-forum.org/mpi3-ft/2011/12/0938.php Looking back at my written notes, they don't have much more than that to add to the discussion. So the problem with (3) seemed to arrise out of the failure handlers, though I am not convinced that they are strictly to blame in this circumstance. It seems that the agreement protocol might be factoring into the discussion as well since it is strongly synchronizing, if not all processes call the operation how does it know when to bail out. The peer processes are (a) calling that operation, (b) going to call it but have not yet, or (c) will never call it because they decided independently not to base on a locally reported process failure. It seems that the core problem has to do with when to break out of the collective creation operation, and when to restore matching. So should re reconsider the restriction on (3)? More to the point, is it safe to not require (3)? -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_Comm_validate - What's in a name?
by Sur, Sayantan 25 Jan '12

25 Jan '12

Hi Josh, I think we also had another discussion that Darius initiated regarding this. We discussed that we will allow p2p communication on communicator with 'holes', but disallow collectives. Since this seemed to satisfy your use cases. We were thinking about bringing in the validate as an "add-on" ticket. Are we still on track to do that? Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Wednesday, January 25, 2012 12:56 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] MPI_Comm_validate - What's in a name? On the call today it was suggested that we re-evaluate the name MPI_Comm_validate. It was pointed out that 'validate' seems a bit too close to 'invalid' which is probably not the semantic that we are trying to imply with the name. An alternative is 'check', but that is a bit close to 'checkpoint' so might not be the best either. So we are looking for a good name. We started a similar discussion for reenable_any_source, which might lend some ideas: http://lists.mpi-forum.org/mpi3-ft/2011/12/0931.php The semantic behind MPI_Comm_validate [at the moment] are: (1) A fault tolerant synchronization point returning a consistent value (failed group) at all participating processes (2) Allow for the posting of new collective operations on the communicator (a communicator with potential holes in it) Name suggestions are welcome. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] MPI_ANY_SOURCE ... again...
by Josh Hursey 25 Jan '12

25 Jan '12

We really need to make a decision on semantics for MPI_ANY_SOURCE. During the plenary session the MPI Forum had a problem with the current proposed semantics. The current proposal states (roughly) that MPI_ANY_SOURCE return when a failure emerges in the communicator. The MPI Forum read this as a strong requirement for -progress- (something the MPI standard tries to stay away from). The alternative proposal is that a receive on MPI_ANY_SOURCE will block until completed with a message. This means that it will -not- return when a new failure has been encountered (even if the calling process is the only process left alive in the communicator). This does get around the concern about progress, but puts a large burden on the end user. There are a couple good use cases for MPI_ANY_SOURCE (grumble, grumble) - Manager/Worker applications, and easy load balancing when multiple incoming messages are expected. This blocking behavior makes the use of MPI_ANY_SOURCE dangerous for fault tolerant applications, and opens up another opportunity for deadlock. For applications that want to use MPI_ANY_SOURCE and be fault tolerant they will need to build their own failure detector on top of MPI using directed point-to-point messages. A basic implementation might post MPI_Irecv()'s to each worker process with an unused tag, then poll on Testany(). If any of these requests complete in error (MPI_ERR_PROC_FAIL_STOP) then the target has failed and the application can take action. This user-level failure detector can (should) be implemented in a third-party library since failure detectors can be difficult to implement in a scalable manner. In reality, the MPI library or the runtime system that supports MPI will already be doing something similar. Even for MPI_ERRORS_ARE_FATAL on MPI_COMM_WORLD, the underlying system must detect the process failure, and terminate all other processes in MPI_COMM_WORLD. So this represents a -detection- of the failure, and a -notification- of the failure throughout the system (though the notification is an order to terminate). For MPI_ERRORS_RETURN, the MPI will use this detection/notification functionality to reason about the state of the message traffic in the system. So it seems silly to force the user to duplicate this (nontrivial) detection/notification functionality on top of MPI, just to avoid the progress discussion. So that is a rough summary of the debate. If we are going to move forward, we need to make a decision on MPI_ANY_SOURCE. I would like to make such a decision before/during the next teleconf (Feb. 1). I'm torn on this one, so I look forward to your comments. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by Josh Hursey 25 Jan '12

25 Jan '12

After discussion on the call it was decided that we will continue with the BLANK semantics. Some notes on the rationale from the discussion: - Though it might be possible to get BLANK from SHRINK, it would be difficult and force applications to use a non-standard third-party library. To get SHRINK from BLANK is trivial - MPI_Comm_split(). - There are a few examples that use SHRINK, but can be just as easily implemented with BLANK. Manager/Worker applications were cited. So a strong use case where SHRINK is the preferred model could not be referenced. - The use case for BLANK is based in applications where the name of the process in the communicator is meaningful in referencing data blocks (e.g., matrix operations). In some of these algorithms a checksum is maintained in a spare column/row. MPI_Reduce is used to generate this value. Though most of these algorithms require a replacement of the missing process, it is not strictly necessary for all failure modes. - A BLANK mode makes it generally easier to reason about process recovery. - SHRINK requires fewer changes to the current MPI standard, but the additional statements to support BLANK are generally non-controversial (we will discuss collectives in another email thread). So after a lengthy discussion it was decided by the group to keep the current model based on a BLANK-like mode. This is based upon both the flexibility of the mode, and assessment of the use cases. Some requested references are at the bottom. There are others, but this should get someone so interested started. -- Josh ---------------------------------------------- Bosilca, G., Delmas, R., Dongarra, J., Langou, J. "Algorithm-based fault tolerance applied to high performance computing" 2008 http://dx.doi.org/10.1016/j.jpdc.2008.12.002 ---------------------------------------------- Engelmann, C., Geist, A. "Super-Scalable Algorithms for Computing on 100,000 Processors" 2005 http://www.springerlink.com/index/10.1007/11428831_39 ---------------------------------------------- Huang, K.H., Abraham, J.A. "Algorithm-Based Fault Tolerance for Matrix Operations" 1984 http://dx.doi.org/10.1109/TC.1984.1676475 ---------------------------------------------- On Wed, Jan 25, 2012 at 5:42 AM, TERRY DONTJE <terry.dontje(a)oracle.com>wrote: > One of the things that I am not enamored with the SHRINK mode is you > cannot implement the BLANK with it but the converse is true. So to me the > BLANK mode seems like a lower level of routines that you can build up from > which seems like a good idea though I admit it might be a little more > complicated but I am not sure it is that much more complicated than SHRINK. > > --td > > > On 1/24/2012 2:41 PM, Graham, Richard L. wrote: > > I will re-iterate what I said before. While this may one mode that apps may want to use, it is not the only mode. In particular, this forces all ranks in a communicator to know about the change, even if they have implemented a "local" algorithm that does not need to know about all failures. > > Rich > > On Jan 24, 2012, at 2:09 PM, Sur, Sayantan wrote: > > Hi Josh, > > Thanks for the crisp characterization of the proposal I was making. It is correct. I was naturally thinking of the SHRINK mode, since it involves least number of changes to MPI standard itself. Folks at the forum also had similar thoughts (e.g. why does MPI_Comm_size() still return count including failed procs). > > Cf. http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT-MPI.pdf > > “4.2 FTMPI_COMM_MODE_SHRINK > > In this communicator mode, the ranks of MPI processes before and after recovery might change, as well as the size of MPI COMM WORLD does change. The appealing part of this communicator mode however is, that all functions specified in MPI-1 and MPI-2 are still valid without any further modification, since groups and communicators do not have wholes (sic) and blank processes.” > > We can discuss further tomorrow as to whether we could go with SHRINK mode (simplifying the proposal). From what I read in the paper, they report being able to convert fault-tolerant master/worker to use both modes. > > Thanks! > > === > Sayantan Sur, Ph.D. > Intel Corp. > > From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]> <mpi3-ft-bounces(a)lists.mpi-forum.org> [mailto:[email protected] <mpi3-ft-bounces(a)lists.mpi-forum.org>] On Behalf Of Josh Hursey > Sent: Tuesday, January 24, 2012 10:39 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal > > First let me say that I greatly appreciate the effort of Sayantan and others to push us towards considering alternative techniques, and stimulating discussion about design decisions. This is exactly the type of discussion that needs to occur, and the working group is the most appropriate place to have it. > > > One of the core suggestions of Sayantan's proposal is the switch from (using FT-MPI's language) a model like BLANK to a model like SHRINK. I think many of the other semantics are derived from this core shift, so we should probably focus the discussion on this point earlier in our conversation. > > > The current RTS proposal allows for a communicator to contain failed processes and continue to be used for all operations, including collectives, after acknowledging them. This matches closely to FT-MPI's BLANK mode. The user can use MPI_Comm_split() to get the equivalent of SHRINK if they need it. > > The suggested modification allows for only/primarily a SHRINK-like mode in order to have full functionality in the communicator. As discussed on the previous call, one can get the BLANK mode by adding a library on top of MPI that virtualizes the communicators to create shadow communicators. The argument for the SHRINK mode is that it is -easier- to pass/explain. > > The reason we chose BLANK was derived from the literature reviewed, code examples available, and feedback from application groups. >From which there seemed to be a strong demand for the BLANK mode. In fact, I had a difficult time finding good use cases for the SHIRNK mode (I'm still looking though). Additionally, a BLANK mode seems also to make it easier to reason about process recovery. To reason about process recovery (something like FT-MPI's REBUILD mode) one needs to be able to reason about the missing processes without changing the identities of the existing processes, which can be difficult in a SHRINK mode. So from this review it seemed that there was an application demand for a BLANK-like mode for the RTS proposal. > > In light of this background, it is concerning to me to advise these application users that MPI will not provide the functionality they require, but they have to depend upon a non-standard, third-party library because we shied away from doing the right thing by them. This background is advised from my review of the state of the art, but others may have alternative evidence/commentary to present as well that could sway the discussion. It just seems like a weak argument that we should do the easy thing at the expense of doing the right thing by the application community. > > > I certainly meant this email to stimulate conversation for the teleconference tomorrow. In particular, I would like those on the list with experience building ABFT/Natural FT applications/libraries (UTK?) to express their perspective on this topic. Hopefully they can help guide us towards the right solution, which might just be a SHRINK-like mode. > > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratoryhttp://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing listmpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> <mpi3-ft(a)lists.mpi-forum.org>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing [email protected]://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > -- > Terry D. Dontje | Principal Software Engineer > Developer Tools Engineering | +1.781.442.2631 > Oracle * - Performance Technologies* > 95 Network Drive, Burlington, MA 01803 > Email terry.dontje(a)oracle.com > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 25 Jan '12

25 Jan '12

Good call today. I think we got the ball rolling. I will be starting email threads shortly on some of the topics discussed on the list. Some items that we will discuss next week (Feb. 1): - Piggybacking - RTS Proposal - Decide on Collectives over communicators with holes - Decide on MPI_ANY_SOURCE semantic - Discuss FailHandlers - Discuss communicator creation matching Then on Thursday, Feb. 2 we will have another teleconf with the RMA group at the usual time. Thanks, Josh On Tue, Jan 24, 2012 at 1:01 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Reminder that we have a teleconf tomorrow, Wednesday, at Noon EST. > > Date: January 25, 2012 > Time: Noon EST/New York > Dial-in information: 877-801-8130 > Code: 1044056 > > > Agenda: > - Suggested modifications to RTS proposal (Sayantan) > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… > http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php > - Tony's alternative model (Tony) > - Do you still need this time? > - Other items? > > > Additional Agenda Items [as time permits]: > - Run-Through Stabilization Proposal > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > - Process Recovery (Josh H.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 > - Piggybacking (Martin S.) > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - MPI Forum notes discussion (?) > http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php > - Error to Fault changes proposal (Darius B.) > http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php > - Network Channel Failure > http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php > - FT Query Interface > http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php > > > Next Meetings: > * Jan. 25, 2012 (Teleconf) > * Feb. 1, 2012 (Teleconf) > * Feb. 8, 2012 (Teleconf - As needed) > * Feb. 15, 2012 (Teleconf) > * Feb. 22, 2012 (Teleconf - As needed) > * Feb. 29, 2012 (Teleconf) > * March 5, 2012 (MPI Forum Meeting) > * March 14, 2012 (Teleconf) > > Respond to the list if you have any additional topics that you would like > to discuss. > > -- Josh > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] MPI_Comm_validate - What's in a name?
by Josh Hursey 25 Jan '12

25 Jan '12

On the call today it was suggested that we re-evaluate the name MPI_Comm_validate. It was pointed out that 'validate' seems a bit too close to 'invalid' which is probably not the semantic that we are trying to imply with the name. An alternative is 'check', but that is a bit close to 'checkpoint' so might not be the best either. So we are looking for a good name. We started a similar discussion for reenable_any_source, which might lend some ideas: http://lists.mpi-forum.org/mpi3-ft/2011/12/0931.php The semantic behind MPI_Comm_validate [at the moment] are: (1) A fault tolerant synchronization point returning a consistent value (failed group) at all participating processes (2) Allow for the posting of new collective operations on the communicator (a communicator with potential holes in it) Name suggestions are welcome. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Fault Tolerance & RMA Discussion
by Josh Hursey 25 Jan '12

25 Jan '12

There was no one date/time that worked for everyone, but I chose a time that worked for most of the respondents. We will meet Thursday, Feb. 2 from 12-1 pm EST/New York to discuss this topic. We can use the following teleconf information: US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 Thanks, Josh On Mon, Jan 23, 2012 at 4:33 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > (Cross posted to both the RMA and FT MPI-3 listservs) > > During the FT plenary session at the Jan. MPI Forum meeting it was > recommended that some of the members of the FT group and the RMA group have > a meeting to hash out the precise details of the FT semantics for the RMA > chapter. So I would like to facilitate such a discussion, preferability in > the next week (so we have time to fine tune things before the next forum > meeting). > > In general, we are trying to answer the question "How should RMA > operations behave when a process failure occurs?" The feeling seemed to be > that the current approach is ok (invalidating the window, forcing > recreation/validation), but the statement that the memory exposed in the > window is 'undefined' seemed excessive. The suggestion was to change the > wording to something like "Only the memory associated with a window that > was targeted by an operation that modified it is undefined after process > failure in the group associated with the window." This lead to a > considerable amount of debate in the meeting, so it was suggested that we > take the discussion offline. > > Below is a link to a doodle poll to find a good time for a teleconf. If > you are interested in participating in this discussion, please fill this > poll out by 2 PM Eastern on Wed. Jan 25 so we can set the date/time. > http://www.doodle.com/vd33va5h8iankega > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by TERRY DONTJE 25 Jan '12

25 Jan '12

One of the things that I am not enamored with the SHRINK mode is you cannot implement the BLANK with it but the converse is true. So to me the BLANK mode seems like a lower level of routines that you can build up from which seems like a good idea though I admit it might be a little more complicated but I am not sure it is that much more complicated than SHRINK. --td On 1/24/2012 2:41 PM, Graham, Richard L. wrote: > I will re-iterate what I said before. While this may one mode that apps may want to use, it is not the only mode. In particular, this forces all ranks in a communicator to know about the change, even if they have implemented a "local" algorithm that does not need to know about all failures. > > Rich > > On Jan 24, 2012, at 2:09 PM, Sur, Sayantan wrote: > > Hi Josh, > > Thanks for the crisp characterization of the proposal I was making. It is correct. I was naturally thinking of the SHRINK mode, since it involves least number of changes to MPI standard itself. Folks at the forum also had similar thoughts (e.g. why does MPI_Comm_size() still return count including failed procs). > > Cf. http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT-MPI.pdf > > ï¿½4.2 FTMPI_COMM_MODE_SHRINK > > In this communicator mode, the ranks of MPI processes before and after recovery might change, as well as the size of MPI COMM WORLD does change. The appealing part of this communicator mode however is, that all functions specified in MPI-1 and MPI-2 are still valid without any further modification, since groups and communicators do not have wholes (sic) and blank processes.ï¿½ > > We can discuss further tomorrow as to whether we could go with SHRINK mode (simplifying the proposal). From what I read in the paper, they report being able to convert fault-tolerant master/worker to use both modes. > > Thanks! > > === > Sayantan Sur, Ph.D. > Intel Corp. > > From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Josh Hursey > Sent: Tuesday, January 24, 2012 10:39 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal > > First let me say that I greatly appreciate the effort of Sayantan and others to push us towards considering alternative techniques, and stimulating discussion about design decisions. This is exactly the type of discussion that needs to occur, and the working group is the most appropriate place to have it. > > > One of the core suggestions of Sayantan's proposal is the switch from (using FT-MPI's language) a model like BLANK to a model like SHRINK. I think many of the other semantics are derived from this core shift, so we should probably focus the discussion on this point earlier in our conversation. > > > The current RTS proposal allows for a communicator to contain failed processes and continue to be used for all operations, including collectives, after acknowledging them. This matches closely to FT-MPI's BLANK mode. The user can use MPI_Comm_split() to get the equivalent of SHRINK if they need it. > > The suggested modification allows for only/primarily a SHRINK-like mode in order to have full functionality in the communicator. As discussed on the previous call, one can get the BLANK mode by adding a library on top of MPI that virtualizes the communicators to create shadow communicators. The argument for the SHRINK mode is that it is -easier- to pass/explain. > > The reason we chose BLANK was derived from the literature reviewed, code examples available, and feedback from application groups.> From which there seemed to be a strong demand for the BLANK mode. In fact, I had a difficult time finding good use cases for the SHIRNK mode (I'm still looking though). Additionally, a BLANK mode seems also to make it easier to reason about process recovery. To reason about process recovery (something like FT-MPI's REBUILD mode) one needs to be able to reason about the missing processes without changing the identities of the existing processes, which can be difficult in a SHRINK mode. So from this review it seemed that there was an application demand for a BLANK-like mode for the RTS proposal. > > In light of this background, it is concerning to me to advise these application users that MPI will not provide the functionality they require, but they have to depend upon a non-standard, third-party library because we shied away from doing the right thing by them. This background is advised from my review of the state of the art, but others may have alternative evidence/commentary to present as well that could sway the discussion. It just seems like a weak argument that we should do the easy thing at the expense of doing the right thing by the application community. > > > I certainly meant this email to stimulate conversation for the teleconference tomorrow. In particular, I would like those on the list with experience building ABFT/Natural FT applications/libraries (UTK?) to express their perspective on this topic. Hopefully they can help guide us towards the right solution, which might just be a SHRINK-like mode. > > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Terry D. Dontje | Principal Software Engineer Developer Tools Engineering | +1.781.442.2631 Oracle *- Performance Technologies* 95 Network Drive, Burlington, MA 01803 Email terry.dontje(a)oracle.com <mailto:[email protected]>

1 0

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by Graham, Richard L. 24 Jan '12

24 Jan '12

Comment in line: On Jan 24, 2012, at 3:11 PM, Sur, Sayantan wrote: >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Graham, Richard L. >> Sent: Tuesday, January 24, 2012 11:42 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS >> proposal >> >> I will re-iterate what I said before. While this may one mode that >> apps may want to use, it is not the only mode. In particular, this >> forces all ranks in a communicator to know about the change, even if >> they have implemented a "local" algorithm that does not need to know >> about all failures. >> > > I guess we can talk about this tomorrow in the WG meeting. My question is the current RTS proposal also requires everyone to call MPI_Comm_validate and know about other failures if they want to use collectives. If you are going to use a global operation, you must make sure all support it, and the validate has been defined as the function to do that. If you don't want global operations, you don't need to do a global check. Also, nothing forces one to call the validate function, but if one wants to know if the collective succeeded on all ranks, one has to call validate - or build the equivalent functionality at the app layer. Rich > > >> Rich >> >> On Jan 24, 2012, at 2:09 PM, Sur, Sayantan wrote: >> >> Hi Josh, >> >> Thanks for the crisp characterization of the proposal I was making. It >> is correct. I was naturally thinking of the SHRINK mode, since it >> involves least number of changes to MPI standard itself. Folks at the >> forum also had similar thoughts (e.g. why does MPI_Comm_size() still >> return count including failed procs). >> >> Cf. http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT- >> MPI.pdf >> >> "4.2 FTMPI_COMM_MODE_SHRINK >> >> In this communicator mode, the ranks of MPI processes before and after >> recovery might change, as well as the size of MPI COMM WORLD does >> change. The appealing part of this communicator mode however is, that >> all functions specified in MPI-1 and MPI-2 are still valid without any >> further modification, since groups and communicators do not have wholes >> (sic) and blank processes." >> >> We can discuss further tomorrow as to whether we could go with SHRINK >> mode (simplifying the proposal). From what I read in the paper, they >> report being able to convert fault-tolerant master/worker to use both >> modes. >> >> Thanks! >> >> === >> Sayantan Sur, Ph.D. >> Intel Corp. >> >> From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org> [mailto:[email protected] >> forum.org] On Behalf Of Josh Hursey >> Sent: Tuesday, January 24, 2012 10:39 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS >> proposal >> >> First let me say that I greatly appreciate the effort of Sayantan and >> others to push us towards considering alternative techniques, and >> stimulating discussion about design decisions. This is exactly the type >> of discussion that needs to occur, and the working group is the most >> appropriate place to have it. >> >> >> One of the core suggestions of Sayantan's proposal is the switch from >> (using FT-MPI's language) a model like BLANK to a model like SHRINK. I >> think many of the other semantics are derived from this core shift, so >> we should probably focus the discussion on this point earlier in our >> conversation. >> >> >> The current RTS proposal allows for a communicator to contain failed >> processes and continue to be used for all operations, including >> collectives, after acknowledging them. This matches closely to FT-MPI's >> BLANK mode. The user can use MPI_Comm_split() to get the equivalent of >> SHRINK if they need it. >> >> The suggested modification allows for only/primarily a SHRINK-like mode >> in order to have full functionality in the communicator. As discussed >> on the previous call, one can get the BLANK mode by adding a library on >> top of MPI that virtualizes the communicators to create shadow >> communicators. The argument for the SHRINK mode is that it is -easier- >> to pass/explain. >> >> The reason we chose BLANK was derived from the literature reviewed, >> code examples available, and feedback from application groups. >From >> which there seemed to be a strong demand for the BLANK mode. In fact, I >> had a difficult time finding good use cases for the SHIRNK mode (I'm >> still looking though). Additionally, a BLANK mode seems also to make it >> easier to reason about process recovery. To reason about process >> recovery (something like FT-MPI's REBUILD mode) one needs to be able to >> reason about the missing processes without changing the identities of >> the existing processes, which can be difficult in a SHRINK mode. So >> from this review it seemed that there was an application demand for a >> BLANK-like mode for the RTS proposal. >> >> In light of this background, it is concerning to me to advise these >> application users that MPI will not provide the functionality they >> require, but they have to depend upon a non-standard, third-party >> library because we shied away from doing the right thing by them. This >> background is advised from my review of the state of the art, but >> others may have alternative evidence/commentary to present as well that >> could sway the discussion. It just seems like a weak argument that we >> should do the easy thing at the expense of doing the right thing by the >> application community. >> >> >> I certainly meant this email to stimulate conversation for the >> teleconference tomorrow. In particular, I would like those on the list >> with experience building ABFT/Natural FT applications/libraries (UTK?) >> to express their perspective on this topic. Hopefully they can help >> guide us towards the right solution, which might just be a SHRINK-like >> mode. >> >> >> -- Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by Sur, Sayantan 24 Jan '12

24 Jan '12

> -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Graham, Richard L. > Sent: Tuesday, January 24, 2012 11:42 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS > proposal > > I will re-iterate what I said before. While this may one mode that > apps may want to use, it is not the only mode. In particular, this > forces all ranks in a communicator to know about the change, even if > they have implemented a "local" algorithm that does not need to know > about all failures. > I guess we can talk about this tomorrow in the WG meeting. My question is the current RTS proposal also requires everyone to call MPI_Comm_validate and know about other failures if they want to use collectives. > Rich > > On Jan 24, 2012, at 2:09 PM, Sur, Sayantan wrote: > > Hi Josh, > > Thanks for the crisp characterization of the proposal I was making. It > is correct. I was naturally thinking of the SHRINK mode, since it > involves least number of changes to MPI standard itself. Folks at the > forum also had similar thoughts (e.g. why does MPI_Comm_size() still > return count including failed procs). > > Cf. http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT- > MPI.pdf > > "4.2 FTMPI_COMM_MODE_SHRINK > > In this communicator mode, the ranks of MPI processes before and after > recovery might change, as well as the size of MPI COMM WORLD does > change. The appealing part of this communicator mode however is, that > all functions specified in MPI-1 and MPI-2 are still valid without any > further modification, since groups and communicators do not have wholes > (sic) and blank processes." > > We can discuss further tomorrow as to whether we could go with SHRINK > mode (simplifying the proposal). From what I read in the paper, they > report being able to convert fault-tolerant master/worker to use both > modes. > > Thanks! > > === > Sayantan Sur, Ph.D. > Intel Corp. > > From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org> [mailto:[email protected] > forum.org] On Behalf Of Josh Hursey > Sent: Tuesday, January 24, 2012 10:39 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS > proposal > > First let me say that I greatly appreciate the effort of Sayantan and > others to push us towards considering alternative techniques, and > stimulating discussion about design decisions. This is exactly the type > of discussion that needs to occur, and the working group is the most > appropriate place to have it. > > > One of the core suggestions of Sayantan's proposal is the switch from > (using FT-MPI's language) a model like BLANK to a model like SHRINK. I > think many of the other semantics are derived from this core shift, so > we should probably focus the discussion on this point earlier in our > conversation. > > > The current RTS proposal allows for a communicator to contain failed > processes and continue to be used for all operations, including > collectives, after acknowledging them. This matches closely to FT-MPI's > BLANK mode. The user can use MPI_Comm_split() to get the equivalent of > SHRINK if they need it. > > The suggested modification allows for only/primarily a SHRINK-like mode > in order to have full functionality in the communicator. As discussed > on the previous call, one can get the BLANK mode by adding a library on > top of MPI that virtualizes the communicators to create shadow > communicators. The argument for the SHRINK mode is that it is -easier- > to pass/explain. > > The reason we chose BLANK was derived from the literature reviewed, > code examples available, and feedback from application groups. >From > which there seemed to be a strong demand for the BLANK mode. In fact, I > had a difficult time finding good use cases for the SHIRNK mode (I'm > still looking though). Additionally, a BLANK mode seems also to make it > easier to reason about process recovery. To reason about process > recovery (something like FT-MPI's REBUILD mode) one needs to be able to > reason about the missing processes without changing the identities of > the existing processes, which can be difficult in a SHRINK mode. So > from this review it seemed that there was an application demand for a > BLANK-like mode for the RTS proposal. > > In light of this background, it is concerning to me to advise these > application users that MPI will not provide the functionality they > require, but they have to depend upon a non-standard, third-party > library because we shied away from doing the right thing by them. This > background is advised from my review of the state of the art, but > others may have alternative evidence/commentary to present as well that > could sway the discussion. It just seems like a weak argument that we > should do the easy thing at the expense of doing the right thing by the > application community. > > > I certainly meant this email to stimulate conversation for the > teleconference tomorrow. In particular, I would like those on the list > with experience building ABFT/Natural FT applications/libraries (UTK?) > to express their perspective on this topic. Hopefully they can help > guide us towards the right solution, which might just be a SHRINK-like > mode. > > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0