February 2009 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Greg Bronevetsky 11 Feb '09

11 Feb '09

I don't understand what you mean by "We can continue to pursue communicator reconstruction interfaces though a virtualization later above MPI." To me it seems that such interfaces will effectively need to implement communicators on top of MPI in order be operational, which will take about as much effort as implementing them inside MPI. In particular, I don't see a way to recreate a communicator using the MPI interface without making collective calls. However, we're defining MPI_Rejoin (or whatever its called) to be a local operation. This means that we cannot use the MPI communicators interface and must instead implement our own communicators. The bottom line is that it does make sense to start implementing support for the FT-MPI model and evolve that to a more elaborate model. However, I don't think that working on the rest above MPI will save us any effort or time. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 01:17 PM 2/11/2009, Josh Hursey wrote: >In our meeting yesterday, I was sitting in the back trying to take in >the complexity of communicator recreation. It seems that much of the >confusion at the moment is that we (at least I) are still not exactly >sure how the interface should be defined and implemented. > >I think of the process fault tolerance specification as a series of >steps that can be individually specified building upon each step while >working towards a specific goal set. From this I was asking myself, is >there any foundational concepts that we can define now so that folks >can start implementation. > >That being said I suggest that we consider FT-MPI's model of all >communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) are >destroyed on a failure as the starting point for implementation. This >would get us started. We can continue to pursue communicator >reconstruction interfaces though a virtualization later above MPI. We >can use this layer to experiment with the communicator recreation >mechanisms in conjunction with applications while pursing the first >step implementation. Once we start to agree on the interface for >communicator reconstruction, then we can start to push it into the MPI >standard/library for a better standard/implementation. > >The communicator virtualization library is a staging area for these >interface ideas that we seem to be struggling with. The virtualization >could be a simple table lookup that matches the Application's Virtual >Communicator Object to the actual MPI Communicator Object that may >have been recreated for you by the virtualization library. > >We should still spend time on talking though usage scenarios for >communicator recreation, since that will eventually be something that >we want to provide to the application. I'm just suggesting that we >specify the first step so we can start experimenting with the >communicator recreation next step. > >What do people think about this as a step forward? > >Best, >Josh >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Communicator Virtualization as a step forward
by Josh Hursey 11 Feb '09

11 Feb '09

In our meeting yesterday, I was sitting in the back trying to take in the complexity of communicator recreation. It seems that much of the confusion at the moment is that we (at least I) are still not exactly sure how the interface should be defined and implemented. I think of the process fault tolerance specification as a series of steps that can be individually specified building upon each step while working towards a specific goal set. From this I was asking myself, is there any foundational concepts that we can define now so that folks can start implementation. That being said I suggest that we consider FT-MPI's model of all communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) are destroyed on a failure as the starting point for implementation. This would get us started. We can continue to pursue communicator reconstruction interfaces though a virtualization later above MPI. We can use this layer to experiment with the communicator recreation mechanisms in conjunction with applications while pursing the first step implementation. Once we start to agree on the interface for communicator reconstruction, then we can start to push it into the MPI standard/library for a better standard/implementation. The communicator virtualization library is a staging area for these interface ideas that we seem to be struggling with. The virtualization could be a simple table lookup that matches the Application's Virtual Communicator Object to the actual MPI Communicator Object that may have been recreated for you by the virtualization library. We should still spend time on talking though usage scenarios for communicator recreation, since that will eventually be something that we want to provide to the application. I'm just suggesting that we specify the first step so we can start experimenting with the communicator recreation next step. What do people think about this as a step forward? Best, Josh

1 0

Re: [Mpi3-ft] Quiescence Interface
by Vinod tipparaju 11 Feb '09

11 Feb '09

Quiecence is certainly a interesting topic, I wasn't able to finish my comments towards the end of the discussion yesterday. One philosophy of a middleware quiecence is to act as a mechanism to tell the middleware to prepare for a recovery, fail if it cannot prepare, and wait until it is told to resume. In the context of a one-sided library by the name ARMCI, a long time ago, I implemented some of the semantics for such a quiecence. What we were trying to accomplish in general there required a un-coordinated global quiecence. What we were discussing yesterday is certainly more general. Yet, a paper discussing the work is at http://domino.research.ibm.com/comm/research_people.nsf/pages/scarpazza.pub… Thanks, Vinod Tipparaju.> From: jjhursey(a)open-mpi.org> Date: Wed, 28 Jan 2009 12:13:44 -0500> To: mpi3-ft(a)lists.mpi-forum.org> Subject: Re: [Mpi3-ft] Quiescence Interface> > Starting discussion on this proposal at the next forum meeting sounds > fine to me. I don't know if we need to devote the whole time to this > proposal, but I can prepare a few things for our discussion. I hope > to have a basic prototype ready by then too.> > I can contact Paul Hargrove (BLCR) to see if he would be interested > in attending this discussion. Since this proposal is a collective > operation, it might be useful to have someone from the collective > working group sit in to make sure the proposal is phrased appropriately.> > Cheers,> Josh> > On Jan 28, 2009, at 10:22 AM, Richard Graham wrote:> > > Josh,> > Thanks for sending this around.> > I would like to schedule a time to start discussion this > > proposal. To do> > so, I think we need to have the impacted parties at the table. > > Since the> > Evergrid (sp) folks and the BLCR folks are in the Bay area, would > > it make> > sense to devote our working group time at the Forum to this > > subject, and> > invite then to attend ? Who else would we want to have participate > > in this> > discussion ?> >> > Thanks,> > Rich> >> >> > On 1/22/09 10:21 AM, "Josh Hursey" <jjhursey(a)open-mpi.org> wrote:> >> >> I have updated the Quiescence Interface proposal on the wiki:> >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Quiescence> >>> >> I am interested in questions and comments from the group regarding > >> the> >> interface, and use-cases. The primary use-case is for application> >> initiated checkpoint/restart, but the semantics of the proposed API> >> likely also open doors to other use-cases.> >>> >> Let me know what you think.> >>> >> Cheers,> >> Josh> >>> >> _______________________________________________> >> mpi3-ft mailing list> >> mpi3-ft(a)lists.mpi-forum.org> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> >> > _______________________________________________> > mpi3-ft mailing list> > mpi3-ft(a)lists.mpi-forum.org> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft> > _______________________________________________> mpi3-ft mailing list> mpi3-ft(a)lists.mpi-forum.org> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Point2Point issue scenario with synchronous notification based on calling communicator only.
by Thomas Herault 10 Feb '09

10 Feb '09

The failure happened in MPI_COMM_WORLD. I assume that for this case, the fault tolerance strategy of library A is based on collective repair. Do you propose to forbid using collective repairs if the user uses such communication scheme? How do you propose to deal with the following example: MPI_COMM_WORLD is the communicator used in library A MPI_COMM_2 is the communicator used in library B rank 0: belongs to MPI_COMM_WORLD *and MPI_COMM_2* -> in library A: do some computation and communications that succeed -> in library A: computes a reversible checksum -> in library A: continue computation, calling other libraries often -> then, at some point in time: -> in library A: MPI_Send(MPI_COMM_WORLD, dst=1); -> crashes -> would have entered library B and here do no communications at this step rank 1: belongs to MPI_COMM_WORLD and MPI_COMM_2 -> in library A: do some computation and communications that succeed -> in library A: computes a reversible checksum -> in library A: continue computation, calling other libraries often -> then, at some point in time, -> in library A: MPI_Recv(MPI_COMM_WORLD, src=0) -> detects the failure -> calls the error manager: collective repair -> would have entered library B and called: MPI_Send(MPI_COMM_2, dst=2); rank 2: belongs to MPI_COMM_WORLD and MPI_COMM_2 -> in library A: do some computation and communications that succeed -> in library A: computes a reversible checksum -> in library A: continue computation, calling other libraries often -> then, at some point in time, -> in library A: does not have to communicate at this step -> in library B: MPI_Recv(MPI_COMM_2, src=1); -> will never succeed To inverse the reversible checksum, the rank 2 needs to enter the collective repair, in order to give to the new rank 0 the data that rank 0 lost with the crash. It is not because the library uses a collective approach for tolerating failures that it needs to do synchronizations all the time when it calls other libraries: if library A highly depends on lower-level libraries, it does not want to synchronize each time it is going to make a call, but it is OK to synchronize for time to time to compute the reversible checksum. Thomas 2009/2/10 Erez Haba <erezh(a)microsoft.com> > Don't' do collective repair in rank 1. Do a non-collective repair (rank 1 > does not require the participation of rank 2 to recover rank 0) > > > > *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *Thomas Herault > *Sent:* Monday, February 09, 2009 4:47 PM > *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > *Subject:* [Mpi3-ft] Point2Point issue scenario with synchronous > notification based on calling communicator only. > > > > Hi list, > > > > with the help of others, here is an adaptation of the "counter-example" > based on p2p communications only. > > > > MPI_COMM_WORLD is the communicator used in library A > > MPI_COMM_2 is the communicator used in library B > > > > rank 0: belongs to MPI_COMM_WORLD only > > -> in library A: MPI_Send(MPI_COMM_WORLD, dst=1); > > -> crashes > > > > rank 1: belongs to MPI_COMM_WORLD and MPI_COMM_2 > > -> in library A: MPI_Recv(MPI_COMM_WORLD, src=0) > > -> detects the failure > > -> calls the error manager: collective repair > > -> would have entere library B and called: MPI_Send(MPI_COMM_2, dst=2); > > > > rank 2: belongs to MPI_COMM_WORLD and MPI_COMM_2 > > -> does nothing in library A except entering library B. > > -> in library B: MPI_Recv(MPI_COMM_2, src=1); > > -> will never succeed > > > > I understand from the discussion we had that a solution would be to > validate COMM_WORLD for process 2 before entering library 2. I agree with > that, but would like you to consider that it virtually means that we ask > users to call a $n^2$ communications operation before any call of any > function of any library (and possibly at the return of calls) if they want > to use collective repairs. I would advocate studying a less > performance-killer approach, where errors of any communicator would be > notified in any MPI call. > > > > Bests, > > Thomas > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] Point2Point issue scenario with synchronous notification based on calling communicator only.
by Erez Haba 10 Feb '09

10 Feb '09

Don't' do collective repair in rank 1. Do a non-collective repair (rank 1 does not require the participation of rank 2 to recover rank 0) From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Thomas Herault Sent: Monday, February 09, 2009 4:47 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Point2Point issue scenario with synchronous notification based on calling communicator only. Hi list, with the help of others, here is an adaptation of the "counter-example" based on p2p communications only. MPI_COMM_WORLD is the communicator used in library A MPI_COMM_2 is the communicator used in library B rank 0: belongs to MPI_COMM_WORLD only -> in library A: MPI_Send(MPI_COMM_WORLD, dst=1); -> crashes rank 1: belongs to MPI_COMM_WORLD and MPI_COMM_2 -> in library A: MPI_Recv(MPI_COMM_WORLD, src=0) -> detects the failure -> calls the error manager: collective repair -> would have entere library B and called: MPI_Send(MPI_COMM_2, dst=2); rank 2: belongs to MPI_COMM_WORLD and MPI_COMM_2 -> does nothing in library A except entering library B. -> in library B: MPI_Recv(MPI_COMM_2, src=1); -> will never succeed I understand from the discussion we had that a solution would be to validate COMM_WORLD for process 2 before entering library 2. I agree with that, but would like you to consider that it virtually means that we ask users to call a $n^2$ communications operation before any call of any function of any library (and possibly at the return of calls) if they want to use collective repairs. I would advocate studying a less performance-killer approach, where errors of any communicator would be notified in any MPI call. Bests, Thomas

1 0

[Mpi3-ft] Point2Point issue scenario with synchronous notification based on calling communicator only.
by Thomas Herault 09 Feb '09

09 Feb '09

Hi list, with the help of others, here is an adaptation of the "counter-example" based on p2p communications only. MPI_COMM_WORLD is the communicator used in library A MPI_COMM_2 is the communicator used in library B rank 0: belongs to MPI_COMM_WORLD only -> in library A: MPI_Send(MPI_COMM_WORLD, dst=1); -> crashes rank 1: belongs to MPI_COMM_WORLD and MPI_COMM_2 -> in library A: MPI_Recv(MPI_COMM_WORLD, src=0) -> detects the failure -> calls the error manager: collective repair -> would have entere library B and called: MPI_Send(MPI_COMM_2, dst=2); rank 2: belongs to MPI_COMM_WORLD and MPI_COMM_2 -> does nothing in library A except entering library B. -> in library B: MPI_Recv(MPI_COMM_2, src=1); -> will never succeed I understand from the discussion we had that a solution would be to validate COMM_WORLD for process 2 before entering library 2. I agree with that, but would like you to consider that it virtually means that we ask users to call a $n^2$ communications operation before any call of any function of any library (and possibly at the return of calls) if they want to use collective repairs. I would advocate studying a less performance-killer approach, where errors of any communicator would be notified in any MPI call. Bests, Thomas

1 0