February 2009 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Greg Bronevetsky 13 Feb '09

13 Feb '09

>Such statements are way to broad to be true. In fact it depends on >what recovery mode was used. Please read the document I sent few >emails ago, to see all the capabilities that FT-MPI provided. I make this statement because the FT-MPI model requires all processes to participate in recovery. At the very least, they need to participate in recreating communicators. This may become quite bad if we're using millions of processes. The problem I have is that this is a property of the model, not one implementation of it. >Again this is not true. First, one will need a kind of database to >store this information (distributed or centralized) that came with its >own scalability and cost problems. In addition, in the context of Good point. The scalability and cost problems will absolutely need to be studied. However, as Thomas also pointed out, layering the current API on top of FT-MPI will be complex and will have very different performance properties, making it useless for such a study. >recovery the new processes will have to retrieve this information and >let everybody else know not only their new contact information but the >fact that they are back in the specified communicator. Unfortunately, >this is [again] _NOT_ a local operation. The fact that you seems to >plan to delegate these problems to the runtime environment, doesn't >make it local nor more scalable. You're right, any implementation would need to do a bunch of additional communication operations in order to support the "local rejoin" API, making it not truly local. However, the key difference is that in FT-MPI the recovery must employ a series of global collective operations that require all processes to synchronize. In contrast, runtime support for the local rejoin option is much less coupled. When a process sends a message to another process, it will need to attach the receiver's expected rank in MPI_COMM_WORLD to the message. If the receiver exists and has the right rank, message delivery occurs fine. If not, the sender gets an error and needs to ask the runtime environment for the correct physical address for the given receiver rank. This system involves probably as much overall communication as the everybody-synchronize approach but since the communication is decoupled, it will have a much smaller hit on performance. One thing I don't know is how to do the above for puts and gets. Is it possible for RDMA hardware to do any verification of the destination process? Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Supalov, Alexander 13 Feb '09

13 Feb '09

Hi, FT must be optional to be accepted. One way is to put it into a subset, say, thru the MPI_INIT_ASSERTED proposal. Another is to add yet one MPI_INIT call that will contain a flag for FT configuration (like MPI_FT_SYNCHRONOUS, MPI_FT_ASYNCHRONOUS, etc.). This was mentioned in relation to the issue notification in the earlier fault handling=error handling proposal of mine, see https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Fault%20Handling. Best regards. Alexander -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Thursday, February 12, 2009 8:50 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >Heh OK. :) Oh well, maybe my statements can then provide some >motivation for users that are interested in FT at very least :). Right, and thanks for the comments! BTW, we are specifically worrying about limiting the effect of the API extensions on performance. However, I'm not sure how we'll deal with people who want to turn FT off. We tossed around the idea of multiple levels of FT support but I'm not sure where we left it. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] Fault recovery of multiple communication libraries
by Thomas Herault 12 Feb '09

12 Feb '09

Sriram, As I understand it, our effort aims at making the MPI library useable after failures, and fault tolerant applications possible above the MPI standard. When mixing multiple communication models, the application will have to do the extra work needed to keep the system view "in sync", as we must do this work in the MPI library to keep MPI processes in sync one with each others. I agree with you that the hardest part, if you are dealing with multiple libraries that tolerate failures, is to coordinate them, by deciding, at the application level, which one is responsible for restarting processes, then do all the necessary work to re-init or mend the other libraries with the new process. In your example, the MPI implementation would restart the failed processes, mend the communicators, obtain all necessary information for mending the ARMCI library (including new hostnames), etc... This is theoretically possible in our proposal. Or do you see something missing? The converse is also possible in our proposal: the ARMCI library could detect the failure, other ways could be used to launch the missing processes, and the new processes would call MPI_Init, and use an MPI_connect / MPI_accept scheme to connect the two wolrds and let the application behave as before the failure. The point for me here is that this is a decision of the fault-tolerant application. Thus, MPI-3.0 should provide what is necessary for the application to do that, but not do it for the application. Do you think there is a missing API / functionality in the current proposal that prevent this kind of scenario? Bests, Thomas Le 12 fï¿½vr. 09 ï¿½ 21:30, Krishnamoorthy, Sriram a ï¿½crit : > Thomas, > > Consider an application that uses both MPI and another communication > library, say ARMCI. The application requires both to be fault- > resilient. > If a failure identified by one library can be notified by the other in > some fashion, both can stay in sync with respect to their view of the > processes and which faults have been observed. This does not have to > collective, with each process having a possibly different view. > > If the communication libraries are not in sync, MPI can handle > failures > whenever an MPI communication is involved. However, the other > communication library might need a mechanism to determine (possibly > without co-operation from other processes) whether the process-view of > the system has changed -- whether a new communicator should be used, a > failed process has been replaced by another one, etc. > > For example, with the FT-MPI-like model where a failed process is > replaced by another, ARMCI would at least need the hostname of the new > process to repair its state. > > Sriram.K > > > ------------------------------ > > Message: 2 > Date: Thu, 12 Feb 2009 16:07:41 -0500 > From: Thomas Herault <herault.thomas(a)gmail.com> > Subject: Re: [Mpi3-ft] Fault recovery of multiple communication > libraries > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group" <mpi3-ft(a)lists.mpi-forum.org> > Message-ID: <6E20C9CC-7CA1-42CA-AB18-55075ACBBED1(a)gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes > > Hello, > > We have not considered mixed communication models yet, and I am not > sure > we need to do so. > Let's call A the process that fails, and B the set of processes that > should be notified. Consider the two following cases: > - A also uses MPI to communicate with processes from B. Then, when a > process from B tries to communicate with A, it will be notified by MPI > of the error > - A does not use MPI to communicate with processes from B. Then, the > application does not need help from MPI to deal with the failure: > MPI is > not broken, thus does not have to be mended. > > Thomas > > > Le 12 f?vr. 09 ? 14:46, Krishnamoorthy, Sriram a ?crit : > >> I would like to understand how MPI can be notified of a failure. >> Consider another communication library (ARMCI/GASnet/...) identifying >> a failure through its own mechanisms. In the current model, how can >> it > >> notify MPI to verify/reconfigure/recover from the error, for example >> by performing an MPI communication to the failed process? >> >> Conversely, can a communication library register to be notified of an >> error that MPI identifies and recovers from, so that that the library >> can take appropriate action? >> >> Sriram.K >> >> > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Fault recovery of multiple communication libraries
by Krishnamoorthy, Sriram 12 Feb '09

12 Feb '09

Thomas, Consider an application that uses both MPI and another communication library, say ARMCI. The application requires both to be fault-resilient. If a failure identified by one library can be notified by the other in some fashion, both can stay in sync with respect to their view of the processes and which faults have been observed. This does not have to collective, with each process having a possibly different view. If the communication libraries are not in sync, MPI can handle failures whenever an MPI communication is involved. However, the other communication library might need a mechanism to determine (possibly without co-operation from other processes) whether the process-view of the system has changed -- whether a new communicator should be used, a failed process has been replaced by another one, etc. For example, with the FT-MPI-like model where a failed process is replaced by another, ARMCI would at least need the hostname of the new process to repair its state. Sriram.K ------------------------------ Message: 2 Date: Thu, 12 Feb 2009 16:07:41 -0500 From: Thomas Herault <herault.thomas(a)gmail.com> Subject: Re: [Mpi3-ft] Fault recovery of multiple communication libraries To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Message-ID: <6E20C9CC-7CA1-42CA-AB18-55075ACBBED1(a)gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Hello, We have not considered mixed communication models yet, and I am not sure we need to do so. Let's call A the process that fails, and B the set of processes that should be notified. Consider the two following cases: - A also uses MPI to communicate with processes from B. Then, when a process from B tries to communicate with A, it will be notified by MPI of the error - A does not use MPI to communicate with processes from B. Then, the application does not need help from MPI to deal with the failure: MPI is not broken, thus does not have to be mended. Thomas Le 12 f?vr. 09 ? 14:46, Krishnamoorthy, Sriram a ?crit : > I would like to understand how MPI can be notified of a failure. > Consider another communication library (ARMCI/GASnet/...) identifying > a failure through its own mechanisms. In the current model, how can it > notify MPI to verify/reconfigure/recover from the error, for example > by performing an MPI communication to the failed process? > > Conversely, can a communication library register to be notified of an > error that MPI identifies and recovers from, so that that the library > can take appropriate action? > > Sriram.K > >

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by George Bosilca 12 Feb '09

12 Feb '09

On Feb 12, 2009, at 14:39 , Greg Bronevetsky wrote: > I don't think that the users are suggesting that they don't want FT > support. It sounds like they just don't value having the ability to > reset the state of the MPI library without having to restart the > applications. Since job schedulers can start up a large-scale > application fairly quickly and they already use global > checkpointing, I'm not surprised that they don't really care about > this. In any case, our plans for the FT spec will allow for more > capability than the FT-MPI spec. FT-MPI is tilted towards global > synchronous recovery solutions, which will have scalability problems > since every process must participate in recovery. Such statements are way to broad to be true. In fact it depends on what recovery mode was used. Please read the document I sent few emails ago, to see all the capabilities that FT-MPI provided. > Our goal with the FT specification is to allow localized recovery as > well. Again this is not true. First, one will need a kind of database to store this information (distributed or centralized) that came with its own scalability and cost problems. In addition, in the context of recovery the new processes will have to retrieve this information and let everybody else know not only their new contact information but the fact that they are back in the specified communicator. Unfortunately, this is [again] _NOT_ a local operation. The fact that you seems to plan to delegate these problems to the runtime environment, doesn't make it local nor more scalable. george. > > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > At 11:16 AM 2/12/2009, Nathan DeBardeleben wrote: >> I really worry about taking the advise of users saying they would >> rather terminate and restart an application than having some >> assistance to help them ride through a problem. If they are >> worried about programming language/model changes, I would encourage >> them to open their eyes. >> Major programming model changes are predicted for > petascale >> computers and even petascale computers are having a hard time with >> classical MPI programming. I think we're more likely to see MPI as >> an underpinning of next-gen models. These users polled might not >> be extreme-scale users, however. >> Working at a laboratory positioning itself for exascale, we are >> intimately aware of the fact that "oh just rerun it" is a worthless >> conclusion. I wish I had more time to assist in this matter but >> our laboratory has cracked down on participation in things that are >> not directly associated with charge codes so it's a bit hard for me >> to spend any sizable amount of time. >> >> Please though, consider the user base when they say things like that. >> I'm sure Rich is well aware of these similar concerns. While MPI >> fault tolerance might not be important to the users running 1000 >> node systems, those of us approaching system mean time to interrupt >> under an hour are quite on the opposite side of that spectrum. >> >> Are the small-system users pushing for FT to not be inside of MPI? >> This is why I was so in favor of some sort of componentized MPI >> where users could exclude FT if they weren't worried about >> reliability (and thereby gain performance) but those of us who were >> in more dangerous reliability regimes could take the performance >> penalty and compile in / load in / configure in / whatever FT. >> >> -- Nathan >> >> --------------------------------------------------------------------- >> Nathan DeBardeleben, Ph.D. >> Los Alamos National Laboratory >> High Performance Computing Systems Integration (HPC-5) >> phone: 505-667-3428 >> email: ndebard(a)lanl.gov >> --------------------------------------------------------------------- >> >> >> Graham, Richard L. wrote: >>> Josh, >>> Very early on in the process we got feedback from users that an >>> ft-mpi like interface was of no interest to them. They would just >>> as soon terminate the application and restart rather than use this >>> sort of approach. Having said that, there is already previous >>> demonstration that the ft-mpi approach is useful for some >>> applications. If you look closely at the spec, the ft-mpi >>> approach is a subset. of the current subset. >>> I am working on pulling out the api's and expanding the >>> explanations. The goal is to have this out before the next >>> telecon in two weeks. >>> Prototyping is under way, with ut, cray, and ornl committed to >>> working on this. Right now supporting infrastructure is being >>> developed. >>> Your point on the mpi 2 interfaces is good. A couple of people >>> had started to look at this when it looked like this might make it >>> into the 2.2 version. The changes seemed to be more extensive >>> than expected, so work stopped. This does need to be picked up on. >>> >>> Rich >>> ------Original Message------ >>> From: Josh Hursey >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group >>> ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control >>> working Group >>> Sent: Feb 12, 2009 8:31 AM >>> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >>> >>> It is a good point that local communicator reconstruction operations >>> require a fundamental change in the way communicators are handled by >>> MPI. With that in mind it would probably take as much effort (if not >>> more) to implement a virtualized version on top of MPI. So maybe it >>> will not help as much as I had originally thought. Outside of the >>> paper, do we have the interface and semantics of these operations >>> described anywhere? I think that would help in trying to keep pace >>> with the use cases. >>> >>> The spirit of the suggestion was as a way to separate what (I think) >>> we can agree on as a first step (FT-MPI-like model) from the >>> communicator reconstruction, which I see as a secondary step. If we >>> stop to write up what the FT-MPI-like model should look like in the >>> standard, then I think we can push forward on other fronts >>> (prototyping of step 1, standardization of step 1, application >>> implementations using step 1) while still trying to figure out how >>> communication reconstruction should be expressed in the standard >>> such >>> that it is usable in target applications. >>> >>> So my motion is that the group explicitly focus effort on writing a >>> document describing the FT-MPI-like model we consider as a >>> foundation. Do so in the MPI standard language, and present it to >>> the >>> MPI Forum for a straw vote in the next couple of meetings. From this >>> document we can continue evolving it to support more advanced >>> features, like communicator reconstruction. >>> >>> I am willing to put effort into making such a document. However, I >>> would like explicit support from the working group in pursing such >>> an >>> effort, and the help of anyone interested in helping write-up/define >>> this specification. >>> >>> So what do people think taking this first step? >>> >>> -- Josh >>> >>> >>> On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: >>> >>> >>>> I don't understand what you mean by "We can continue to pursue >>>> communicator reconstruction interfaces though a virtualization >>>> later above MPI." To me it seems that such interfaces will >>>> effectively need to implement communicators on top of MPI in order >>>> be operational, which will take about as much effort as >>>> implementing them inside MPI. In particular, I don't see a way to >>>> recreate a communicator using the MPI interface without making >>>> collective calls. However, we're defining MPI_Rejoin (or whatever >>>> its called) to be a local operation. This means that we cannot use >>>> the MPI communicators interface and must instead implement our own >>>> communicators. >>>> >>>> The bottom line is that it does make sense to start implementing >>>> support for the FT-MPI model and evolve that to a more elaborate >>>> model. However, I don't think that working on the rest above MPI >>>> will save us any effort or time. >>>> >>>> Greg Bronevetsky >>>> Post-Doctoral Researcher >>>> 1028 Building 451 >>>> Lawrence Livermore National Lab >>>> (925) 424-5756 >>>> bronevetsky1(a)llnl.gov >>>> >>>> At 01:17 PM 2/11/2009, Josh Hursey wrote: >>>> >>>>> In our meeting yesterday, I was sitting in the back trying to >>>>> take in >>>>> the complexity of communicator recreation. It seems that much of >>>>> the >>>>> confusion at the moment is that we (at least I) are still not >>>>> exactly >>>>> sure how the interface should be defined and implemented. >>>>> >>>>> I think of the process fault tolerance specification as a series >>>>> of >>>>> steps that can be individually specified building upon each step >>>>> while >>>>> working towards a specific goal set. From this I was asking >>>>> myself, is >>>>> there any foundational concepts that we can define now so that >>>>> folks >>>>> can start implementation. >>>>> >>>>> That being said I suggest that we consider FT-MPI's model of all >>>>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>>>> are >>>>> destroyed on a failure as the starting point for implementation. >>>>> This >>>>> would get us started. We can continue to pursue communicator >>>>> reconstruction interfaces though a virtualization later above >>>>> MPI. We >>>>> can use this layer to experiment with the communicator recreation >>>>> mechanisms in conjunction with applications while pursing the >>>>> first >>>>> step implementation. Once we start to agree on the interface for >>>>> communicator reconstruction, then we can start to push it into the >>>>> MPI >>>>> standard/library for a better standard/implementation. >>>>> >>>>> The communicator virtualization library is a staging area for >>>>> these >>>>> interface ideas that we seem to be struggling with. The >>>>> virtualization >>>>> >>> >>> ------Original Message Truncated------ >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Fault recovery of multiple communication libraries
by Thomas Herault 12 Feb '09

12 Feb '09

Hello, We have not considered mixed communication models yet, and I am not sure we need to do so. Let's call A the process that fails, and B the set of processes that should be notified. Consider the two following cases: - A also uses MPI to communicate with processes from B. Then, when a process from B tries to communicate with A, it will be notified by MPI of the error - A does not use MPI to communicate with processes from B. Then, the application does not need help from MPI to deal with the failure: MPI is not broken, thus does not have to be mended. Thomas Le 12 fï¿½vr. 09 ï¿½ 14:46, Krishnamoorthy, Sriram a ï¿½crit : > I would like to understand how MPI can be notified of a failure. > Consider another communication library (ARMCI/GASnet/...) > identifying a > failure through its own mechanisms. In the current model, how can it > notify MPI to verify/reconfigure/recover from the error, for example > by > performing an MPI communication to the failed process? > > Conversely, can a communication library register to be notified of an > error that MPI identifies and recovers from, so that that the library > can take appropriate action? > > Sriram.K > > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of > mpi3-ft-request(a)lists.mpi-forum.org > Sent: Thursday, February 12, 2009 11:17 AM > To: mpi3-ft(a)lists.mpi-forum.org > Subject: mpi3-ft Digest, Vol 13, Issue 4 > > Send mpi3-ft mailing list submissions to > mpi3-ft(a)lists.mpi-forum.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > or, via email, send a message with subject or body 'help' to > mpi3-ft-request(a)lists.mpi-forum.org > > You can reach the person managing the list at > mpi3-ft-owner(a)lists.mpi-forum.org > > When replying, please edit your Subject line so it is more specific > than > "Re: Contents of mpi3-ft digest..." > > > Today's Topics: > > 1. Re: Communicator Virtualization as a step forward (Josh Hursey) > 2. Re: Communicator Virtualization as a step forward > (Graham, Richard L.) > 3. Re: Communicator Virtualization as a step forward (George > Bosilca) > 4. Re: Communicator Virtualization as a step forward > (Nathan DeBardeleben) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 12 Feb 2009 12:23:49 -0500 > From: Josh Hursey <jjhursey(a)open-mpi.org> > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group" <mpi3-ft(a)lists.mpi-forum.org> > Message-ID: <4554DA1F-BEF7-4F03-8B1C-5B5BF2783477(a)open-mpi.org> > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > > Yeah I was planning on using that document as a starting point. I > wanted > to look over it again and see if anything needs to change given some > of > the discussions that we have been having in the group. It may also > need > some additional language about MPI2 interfaces. It has been while > since > I have looked over this particular document. I would also like to add > some more control for process startup, but we may decide to take > that on > as a secondary step. > > Does UTK still have the LaTeX for that document somewhere? Do you know > if Graham would be interested in participating in this development? > > Cheers, > Josh > > On Feb 12, 2009, at 10:43 AM, George Bosilca wrote: > >> Josh, the document that you talk about already exist. It was >> published > >> in ISC'04. Here is the >> link:http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004- >> FT-M >> PI.pdf >> >> george. > > > > ------------------------------ > > Message: 2 > Date: Thu, 12 Feb 2009 14:01:15 -0500 > From: "Graham, Richard L." <rlgraham(a)ornl.gov> > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > To: mpi3-ft(a)lists.mpi-forum.org > Message-ID: > <537C6C0940C6C143AA46A88946B854170F0FC367(a)ORNLEXCHANGE.ornl.gov> > Content-Type: text/plain; charset=UTF-8 > > Josh, > Very early on in the process we got feedback from users that an ft- > mpi > like interface was of no interest to them. They would just as soon > terminate the application and restart rather than use this sort of > approach. Having said that, there is already previous demonstration > that the ft-mpi approach is useful for some applications. If you look > closely at the spec, the ft-mpi approach is a subset. of the current > subset. > I am working on pulling out the api's and expanding the explanations. > The goal is to have this out before the next telecon in two weeks. > Prototyping is under way, with ut, cray, and ornl committed to > working > on this. Right now supporting infrastructure is being developed. > Your point on the mpi 2 interfaces is good. A couple of people had > started to look at this when it looked like this might make it into > the > 2.2 version. The changes seemed to be more extensive than expected, > so > work stopped. This does need to be picked up on. > > Rich > ------Original Message------ > From: Josh Hursey > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group > Sent: Feb 12, 2009 8:31 AM > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > It is a good point that local communicator reconstruction operations > require a fundamental change in the way communicators are handled by > MPI. With that in mind it would probably take as much effort (if not > more) to implement a virtualized version on top of MPI. So maybe it > will > not help as much as I had originally thought. Outside of the paper, do > we have the interface and semantics of these operations described > anywhere? I think that would help in trying to keep pace with the use > cases. > > The spirit of the suggestion was as a way to separate what (I think) > we > can agree on as a first step (FT-MPI-like model) from the communicator > reconstruction, which I see as a secondary step. If we stop to write > up > what the FT-MPI-like model should look like in the standard, then I > think we can push forward on other fronts (prototyping of step 1, > standardization of step 1, application implementations using step 1) > while still trying to figure out how communication reconstruction > should > be expressed in the standard such that it is usable in target > applications. > > So my motion is that the group explicitly focus effort on writing a > document describing the FT-MPI-like model we consider as a foundation. > Do so in the MPI standard language, and present it to the MPI Forum > for > a straw vote in the next couple of meetings. From this document we can > continue evolving it to support more advanced features, like > communicator reconstruction. > > I am willing to put effort into making such a document. However, I > would > like explicit support from the working group in pursing such an > effort, > and the help of anyone interested in helping write-up/define this > specification. > > So what do people think taking this first step? > > -- Josh > > > On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: > >> I don't understand what you mean by "We can continue to pursue >> communicator reconstruction interfaces though a virtualization later >> above MPI." To me it seems that such interfaces will effectively >> need > >> to implement communicators on top of MPI in order be operational, >> which will take about as much effort as implementing them inside MPI. >> In particular, I don't see a way to recreate a communicator using the >> MPI interface without making collective calls. However, we're >> defining > >> MPI_Rejoin (or whatever its called) to be a local operation. This >> means that we cannot use the MPI communicators interface and must >> instead implement our own communicators. >> >> The bottom line is that it does make sense to start implementing >> support for the FT-MPI model and evolve that to a more elaborate >> model. However, I don't think that working on the rest above MPI will >> save us any effort or time. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> >> At 01:17 PM 2/11/2009, Josh Hursey wrote: >>> In our meeting yesterday, I was sitting in the back trying to take >>> in > >>> the complexity of communicator recreation. It seems that much of the >>> confusion at the moment is that we (at least I) are still not >>> exactly > >>> sure how the interface should be defined and implemented. >>> >>> I think of the process fault tolerance specification as a series of >>> steps that can be individually specified building upon each step >>> while working towards a specific goal set. From this I was asking >>> myself, is there any foundational concepts that we can define now so >>> that folks can start implementation. >>> >>> That being said I suggest that we consider FT-MPI's model of all >>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>> are destroyed on a failure as the starting point for implementation. >>> This would get us started. We can continue to pursue communicator >>> reconstruction interfaces though a virtualization later above MPI. >>> We > >>> can use this layer to experiment with the communicator recreation >>> mechanisms in conjunction with applications while pursing the first >>> step implementation. Once we start to agree on the interface for >>> communicator reconstruction, then we can start to push it into the >>> MPI standard/library for a better standard/implementation. >>> >>> The communicator virtualization library is a staging area for these >>> interface ideas that we seem to be struggling with. The >>> virtualization > > ------Original Message Truncated------ > > > > ------------------------------ > > Message: 3 > Date: Thu, 12 Feb 2009 14:16:09 -0500 > From: George Bosilca <bosilca(a)eecs.utk.edu> > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group" <mpi3-ft(a)lists.mpi-forum.org> > Message-ID: <CD306C06-5662-4060-9E95-255852FE7BB1(a)eecs.utk.edu> > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > > I don't necessarily agree with the statement that FT-MPI is a subset > of > the current spec. As the current spec can be implemented on top of > FT-MPI (with help from the PMPI interface), this tend to prove the > opposite. > > However, I agree there are several features in the current spec that > were not covered by the FT-MPI spec, but these features can be > implemented on top of FT-MPI. As far as I understood, this is what > Josh > proposed, as this will give a quick start (i.e. FT-MPI > implementation is > already available). > > george. > > On Feb 12, 2009, at 14:01 , Graham, Richard L. wrote: > >> Josh, >> Very early on in the process we got feedback from users that an ft- >> mpi like interface was of no interest to them. They would just as >> soon terminate the application and restart rather than use this sort >> of approach. Having said that, there is already previous >> demonstration that the ft-mpi approach is useful for some >> applications. If you look closely at the spec, the ft-mpi approach >> is > >> a subset. of the current subset. >> I am working on pulling out the api's and expanding the explanations. > >> The goal is to have this out before the next telecon in two weeks. >> Prototyping is under way, with ut, cray, and ornl committed to >> working on this. Right now supporting infrastructure is being >> developed. >> Your point on the mpi 2 interfaces is good. A couple of people had >> started to look at this when it looked like this might make it into >> the 2.2 version. The changes seemed to be more extensive than >> expected, so work stopped. This does need to be picked up on. >> >> Rich >> ------Original Message------ >> From: Josh Hursey >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working >> Group >> Sent: Feb 12, 2009 8:31 AM >> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >> >> It is a good point that local communicator reconstruction operations >> require a fundamental change in the way communicators are handled by >> MPI. With that in mind it would probably take as much effort (if not >> more) to implement a virtualized version on top of MPI. So maybe it >> will not help as much as I had originally thought. Outside of the >> paper, do we have the interface and semantics of these operations >> described anywhere? I think that would help in trying to keep pace >> with the use cases. >> >> The spirit of the suggestion was as a way to separate what (I think) >> we can agree on as a first step (FT-MPI-like model) from the >> communicator reconstruction, which I see as a secondary step. If we >> stop to write up what the FT-MPI-like model should look like in the >> standard, then I think we can push forward on other fronts >> (prototyping of step 1, standardization of step 1, application >> implementations using step 1) while still trying to figure out how >> communication reconstruction should be expressed in the standard such >> that it is usable in target applications. >> >> So my motion is that the group explicitly focus effort on writing a >> document describing the FT-MPI-like model we consider as a >> foundation. > >> Do so in the MPI standard language, and present it to the MPI Forum >> for a straw vote in the next couple of meetings. From this document >> we > >> can continue evolving it to support more advanced features, like >> communicator reconstruction. >> >> I am willing to put effort into making such a document. However, I >> would like explicit support from the working group in pursing such an >> effort, and the help of anyone interested in helping write-up/define >> this specification. >> >> So what do people think taking this first step? >> >> -- Josh >> >> >> On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: >> >>> I don't understand what you mean by "We can continue to pursue >>> communicator reconstruction interfaces though a virtualization later >>> above MPI." To me it seems that such interfaces will effectively >>> need to implement communicators on top of MPI in order be >>> operational, which will take about as much effort as implementing >>> them inside MPI. In particular, I don't see a way to recreate a >>> communicator using the MPI interface without making collective >>> calls. > >>> However, we're defining MPI_Rejoin (or whatever its called) to be a >>> local operation. This means that we cannot use the MPI communicators >>> interface and must instead implement our own communicators. >>> >>> The bottom line is that it does make sense to start implementing >>> support for the FT-MPI model and evolve that to a more elaborate >>> model. However, I don't think that working on the rest above MPI >>> will > >>> save us any effort or time. >>> >>> Greg Bronevetsky >>> Post-Doctoral Researcher >>> 1028 Building 451 >>> Lawrence Livermore National Lab >>> (925) 424-5756 >>> bronevetsky1(a)llnl.gov >>> >>> At 01:17 PM 2/11/2009, Josh Hursey wrote: >>>> In our meeting yesterday, I was sitting in the back trying to take >>>> in the complexity of communicator recreation. It seems that much of >>>> the confusion at the moment is that we (at least I) are still not >>>> exactly sure how the interface should be defined and implemented. >>>> >>>> I think of the process fault tolerance specification as a series of >>>> steps that can be individually specified building upon each step >>>> while working towards a specific goal set. From this I was asking >>>> myself, is there any foundational concepts that we can define now >>>> so > >>>> that folks can start implementation. >>>> >>>> That being said I suggest that we consider FT-MPI's model of all >>>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>>> are destroyed on a failure as the starting point for >>>> implementation. >>>> This >>>> would get us started. We can continue to pursue communicator >>>> reconstruction interfaces though a virtualization later above MPI. >>>> We >>>> can use this layer to experiment with the communicator recreation >>>> mechanisms in conjunction with applications while pursing the first >>>> step implementation. Once we start to agree on the interface for >>>> communicator reconstruction, then we can start to push it into the >>>> MPI standard/library for a better standard/implementation. >>>> >>>> The communicator virtualization library is a staging area for these >>>> interface ideas that we seem to be struggling with. The >>>> virtualization >> >> ------Original Message Truncated------ >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > ------------------------------ > > Message: 4 > Date: Thu, 12 Feb 2009 12:16:23 -0700 > From: Nathan DeBardeleben <ndebard(a)lanl.gov> > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group" <mpi3-ft(a)lists.mpi-forum.org> > Message-ID: <49947587.6070604(a)lanl.gov> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > I really worry about taking the advise of users saying they would > rather > terminate and restart an application than having some assistance to > help > them ride through a problem. If they are worried about programming > language/model changes, I would encourage them to open their eyes. > Major programming model changes are predicted for > petascale > computers > and even petascale computers are having a hard time with classical MPI > programming. I think we're more likely to see MPI as an > underpinning of > next-gen models. These users polled might not be extreme-scale users, > however. > > Working at a laboratory positioning itself for exascale, we are > intimately aware of the fact that "oh just rerun it" is a worthless > conclusion. I wish I had more time to assist in this matter but our > laboratory has cracked down on participation in things that are not > directly associated with charge codes so it's a bit hard for me to > spend > any sizable amount of time. > > Please though, consider the user base when they say things like that. > I'm sure Rich is well aware of these similar concerns. While MPI > fault > tolerance might not be important to the users running 1000 node > systems, > those of us approaching system mean time to interrupt under an hour > are > quite on the opposite side of that spectrum. > > Are the small-system users pushing for FT to not be inside of MPI? > This > is why I was so in favor of some sort of componentized MPI where users > could exclude FT if they weren't worried about reliability (and > thereby > gain performance) but those of us who were in more dangerous > reliability > regimes could take the performance penalty and compile in / load in / > configure in / whatever FT. > > -- Nathan > > --------------------------------------------------------------------- > Nathan DeBardeleben, Ph.D. > Los Alamos National Laboratory > High Performance Computing Systems Integration (HPC-5) > phone: 505-667-3428 > email: ndebard(a)lanl.gov > --------------------------------------------------------------------- > > > > Graham, Richard L. wrote: >> Josh, >> Very early on in the process we got feedback from users that an > ft-mpi like interface was of no interest to them. They would just as > soon terminate the application and restart rather than use this sort > of > approach. Having said that, there is already previous demonstration > that the ft-mpi approach is useful for some applications. If you look > closely at the spec, the ft-mpi approach is a subset. of the current > subset. >> I am working on pulling out the api's and expanding the > explanations. The goal is to have this out before the next telecon in > two weeks. >> Prototyping is under way, with ut, cray, and ornl committed to > working on this. Right now supporting infrastructure is being > developed. >> Your point on the mpi 2 interfaces is good. A couple of people had > started to look at this when it looked like this might make it into > the > 2.2 version. The changes seemed to be more extensive than expected, > so > work stopped. This does need to be picked up on. >> >> Rich >> ------Original Message------ >> From: Josh Hursey >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working >> Group >> Sent: Feb 12, 2009 8:31 AM >> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >> >> It is a good point that local communicator reconstruction operations >> require a fundamental change in the way communicators are handled by >> MPI. With that in mind it would probably take as much effort (if not >> more) to implement a virtualized version on top of MPI. So maybe it >> will not help as much as I had originally thought. Outside of the >> paper, do we have the interface and semantics of these operations >> described anywhere? I think that would help in trying to keep pace >> with the use cases. >> >> The spirit of the suggestion was as a way to separate what (I think) >> we can agree on as a first step (FT-MPI-like model) from the >> communicator reconstruction, which I see as a secondary step. If we >> stop to write up what the FT-MPI-like model should look like in the >> standard, then I think we can push forward on other fronts >> (prototyping of step 1, standardization of step 1, application >> implementations using step 1) while still trying to figure out how >> communication reconstruction should be expressed in the standard such >> that it is usable in target applications. >> >> So my motion is that the group explicitly focus effort on writing a >> document describing the FT-MPI-like model we consider as a >> foundation. > >> Do so in the MPI standard language, and present it to the MPI Forum >> for a straw vote in the next couple of meetings. From this document >> we > >> can continue evolving it to support more advanced features, like >> communicator reconstruction. >> >> I am willing to put effort into making such a document. However, I >> would like explicit support from the working group in pursing such an >> effort, and the help of anyone interested in helping write-up/define >> this specification. >> >> So what do people think taking this first step? >> >> -- Josh >> >> >> On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: >> >> >>> I don't understand what you mean by "We can continue to pursue >>> communicator reconstruction interfaces though a virtualization later >>> above MPI." To me it seems that such interfaces will effectively >>> need to implement communicators on top of MPI in order be >>> operational, which will take about as much effort as implementing >>> them inside MPI. In particular, I don't see a way to recreate a >>> communicator using the MPI interface without making collective >>> calls. > >>> However, we're defining MPI_Rejoin (or whatever its called) to be a >>> local operation. This means that we cannot use the MPI communicators >>> interface and must instead implement our own communicators. >>> >>> The bottom line is that it does make sense to start implementing >>> support for the FT-MPI model and evolve that to a more elaborate >>> model. However, I don't think that working on the rest above MPI >>> will > >>> save us any effort or time. >>> >>> Greg Bronevetsky >>> Post-Doctoral Researcher >>> 1028 Building 451 >>> Lawrence Livermore National Lab >>> (925) 424-5756 >>> bronevetsky1(a)llnl.gov >>> >>> At 01:17 PM 2/11/2009, Josh Hursey wrote: >>> >>>> In our meeting yesterday, I was sitting in the back trying to take >>>> in the complexity of communicator recreation. It seems that much of >>>> the confusion at the moment is that we (at least I) are still not >>>> exactly sure how the interface should be defined and implemented. >>>> >>>> I think of the process fault tolerance specification as a series of >>>> steps that can be individually specified building upon each step >>>> while working towards a specific goal set. From this I was asking >>>> myself, is there any foundational concepts that we can define now >>>> so > >>>> that folks can start implementation. >>>> >>>> That being said I suggest that we consider FT-MPI's model of all >>>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>>> are destroyed on a failure as the starting point for >>>> implementation. > >>>> This would get us started. We can continue to pursue communicator >>>> reconstruction interfaces though a virtualization later above MPI. >>>> We can use this layer to experiment with the communicator >>>> recreation > >>>> mechanisms in conjunction with applications while pursing the first >>>> step implementation. Once we start to agree on the interface for >>>> communicator reconstruction, then we can start to push it into the >>>> MPI standard/library for a better standard/implementation. >>>> >>>> The communicator virtualization library is a staging area for these >>>> interface ideas that we seem to be struggling with. The >>>> virtualization >>>> >> >> ------Original Message Truncated------ >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > ------------------------------ > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > End of mpi3-ft Digest, Vol 13, Issue 4 > ************************************** > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by George Bosilca 12 Feb '09

12 Feb '09

The way to implement this was outlined in my previous email. As I stated there, in FT-MPI all remaining processes are aware of the fault, and therefore they can all participate in recreating the communicators. george. On Feb 12, 2009, at 14:33 , Greg Bronevetsky wrote: > You can't just implement the new spec on top of FT-MPI. There is no > way for MPI_Rejoin to be a local operation since we can only use > collective MPI communicator creation functions. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > At 11:16 AM 2/12/2009, George Bosilca wrote: >> I don't necessarily agree with the statement that FT-MPI is a subset >> of the current spec. As the current spec can be implemented on top of >> FT-MPI (with help from the PMPI interface), this tend to prove the >> opposite. >> >> However, I agree there are several features in the current spec that >> were not covered by the FT-MPI spec, but these features can be >> implemented on top of FT-MPI. As far as I understood, this is what >> Josh proposed, as this will give a quick start (i.e. FT-MPI >> implementation is already available). >> >> george. >> >> On Feb 12, 2009, at 14:01 , Graham, Richard L. wrote: >> >>> Josh, >>> Very early on in the process we got feedback from users that an >>> ft- mpi like interface was of no interest to them. They would >>> just as >>> soon terminate the application and restart rather than use this sort >>> of approach. Having said that, there is already previous >>> demonstration that the ft-mpi approach is useful for some >>> applications. If you look closely at the spec, the ft-mpi approach >>> is a subset. of the current subset. >>> I am working on pulling out the api's and expanding the >>> explanations. The goal is to have this out before the next telecon >>> in two weeks. >>> Prototyping is under way, with ut, cray, and ornl committed to >>> working on this. Right now supporting infrastructure is being >>> developed. >>> Your point on the mpi 2 interfaces is good. A couple of people had >>> started to look at this when it looked like this might make it into >>> the 2.2 version. The changes seemed to be more extensive than >>> expected, so work stopped. This does need to be picked up on. >>> >>> Rich >>> ------Original Message------ >>> From: Josh Hursey >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group >>> ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group >>> Sent: Feb 12, 2009 8:31 AM >>> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >>> >>> It is a good point that local communicator reconstruction operations >>> require a fundamental change in the way communicators are handled by >>> MPI. With that in mind it would probably take as much effort (if not >>> more) to implement a virtualized version on top of MPI. So maybe it >>> will not help as much as I had originally thought. Outside of the >>> paper, do we have the interface and semantics of these operations >>> described anywhere? I think that would help in trying to keep pace >>> with the use cases. >>> >>> The spirit of the suggestion was as a way to separate what (I think) >>> we can agree on as a first step (FT-MPI-like model) from the >>> communicator reconstruction, which I see as a secondary step. If we >>> stop to write up what the FT-MPI-like model should look like in the >>> standard, then I think we can push forward on other fronts >>> (prototyping of step 1, standardization of step 1, application >>> implementations using step 1) while still trying to figure out how >>> communication reconstruction should be expressed in the standard >>> such >>> that it is usable in target applications. >>> >>> So my motion is that the group explicitly focus effort on writing a >>> document describing the FT-MPI-like model we consider as a >>> foundation. Do so in the MPI standard language, and present it to >>> the >>> MPI Forum for a straw vote in the next couple of meetings. From this >>> document we can continue evolving it to support more advanced >>> features, like communicator reconstruction. >>> >>> I am willing to put effort into making such a document. However, I >>> would like explicit support from the working group in pursing such >>> an >>> effort, and the help of anyone interested in helping write-up/define >>> this specification. >>> >>> So what do people think taking this first step? >>> >>> -- Josh >>> >>> >>> On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: >>> >>>> I don't understand what you mean by "We can continue to pursue >>>> communicator reconstruction interfaces though a virtualization >>>> later above MPI." To me it seems that such interfaces will >>>> effectively need to implement communicators on top of MPI in order >>>> be operational, which will take about as much effort as >>>> implementing them inside MPI. In particular, I don't see a way to >>>> recreate a communicator using the MPI interface without making >>>> collective calls. However, we're defining MPI_Rejoin (or whatever >>>> its called) to be a local operation. This means that we cannot use >>>> the MPI communicators interface and must instead implement our own >>>> communicators. >>>> >>>> The bottom line is that it does make sense to start implementing >>>> support for the FT-MPI model and evolve that to a more elaborate >>>> model. However, I don't think that working on the rest above MPI >>>> will save us any effort or time. >>>> >>>> Greg Bronevetsky >>>> Post-Doctoral Researcher >>>> 1028 Building 451 >>>> Lawrence Livermore National Lab >>>> (925) 424-5756 >>>> bronevetsky1(a)llnl.gov >>>> >>>> At 01:17 PM 2/11/2009, Josh Hursey wrote: >>>>> In our meeting yesterday, I was sitting in the back trying to take >>>>> in >>>>> the complexity of communicator recreation. It seems that much of >>>>> the >>>>> confusion at the moment is that we (at least I) are still not >>>>> exactly >>>>> sure how the interface should be defined and implemented. >>>>> >>>>> I think of the process fault tolerance specification as a series >>>>> of >>>>> steps that can be individually specified building upon each step >>>>> while >>>>> working towards a specific goal set. From this I was asking >>>>> myself, is >>>>> there any foundational concepts that we can define now so that >>>>> folks >>>>> can start implementation. >>>>> >>>>> That being said I suggest that we consider FT-MPI's model of all >>>>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>>>> are >>>>> destroyed on a failure as the starting point for implementation. >>>>> This >>>>> would get us started. We can continue to pursue communicator >>>>> reconstruction interfaces though a virtualization later above MPI. >>>>> We >>>>> can use this layer to experiment with the communicator recreation >>>>> mechanisms in conjunction with applications while pursing the >>>>> first >>>>> step implementation. Once we start to agree on the interface for >>>>> communicator reconstruction, then we can start to push it into the >>>>> MPI >>>>> standard/library for a better standard/implementation. >>>>> >>>>> The communicator virtualization library is a staging area for >>>>> these >>>>> interface ideas that we seem to be struggling with. The >>>>> virtualization >>> >>> ------Original Message Truncated------ >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Thomas Herault 12 Feb '09

12 Feb '09

I'm not so sure about that. You can conceive an emulator of the new spec above the collective error handler of FT-MPI: of course, in the real execution, when an error occurs, all living processes would have to enter the error handler to mend the MPI_COMM_WORLD, and then rebuild the different communicators that existed. However, the emulator can hide this collective operation from the application by re-entering the communication, and then only consider if the communication should fail (because it involves a failed communicator), or not (because the communicator should not be informed of errors in the model we propose). My main concern with this approach is that the impact on performances would be tremendous, and I'm not sure that the (performance) results obtained with the emulator can give any form of insight on the performances a good implementation would do. It could however serve as a proof of concept for users to try the new model we propose, without waiting for a complete new implementation. I'm not sure either which would be more hard to do: the emulator or the real implementation... Thomas Le 12 fï¿½vr. 09 ï¿½ 14:33, Greg Bronevetsky a ï¿½crit : > You can't just implement the new spec on top of FT-MPI. There is no > way for MPI_Rejoin to be a local operation since we can only use > collective MPI communicator creation functions. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > At 11:16 AM 2/12/2009, George Bosilca wrote: >> I don't necessarily agree with the statement that FT-MPI is a subset >> of the current spec. As the current spec can be implemented on top of >> FT-MPI (with help from the PMPI interface), this tend to prove the >> opposite. >> >> However, I agree there are several features in the current spec that >> were not covered by the FT-MPI spec, but these features can be >> implemented on top of FT-MPI. As far as I understood, this is what >> Josh proposed, as this will give a quick start (i.e. FT-MPI >> implementation is already available). >> >> george. >> >> On Feb 12, 2009, at 14:01 , Graham, Richard L. wrote: >> >>> Josh, >>> Very early on in the process we got feedback from users that an >>> ft- mpi like interface was of no interest to them. They would >>> just as >>> soon terminate the application and restart rather than use this sort >>> of approach. Having said that, there is already previous >>> demonstration that the ft-mpi approach is useful for some >>> applications. If you look closely at the spec, the ft-mpi approach >>> is a subset. of the current subset. >>> I am working on pulling out the api's and expanding the >>> explanations. The goal is to have this out before the next telecon >>> in two weeks. >>> Prototyping is under way, with ut, cray, and ornl committed to >>> working on this. Right now supporting infrastructure is being >>> developed. >>> Your point on the mpi 2 interfaces is good. A couple of people had >>> started to look at this when it looked like this might make it into >>> the 2.2 version. The changes seemed to be more extensive than >>> expected, so work stopped. This does need to be picked up on. >>> >>> Rich >>> ------Original Message------ >>> From: Josh Hursey >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group >>> ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group >>> Sent: Feb 12, 2009 8:31 AM >>> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >>> >>> It is a good point that local communicator reconstruction operations >>> require a fundamental change in the way communicators are handled by >>> MPI. With that in mind it would probably take as much effort (if not >>> more) to implement a virtualized version on top of MPI. So maybe it >>> will not help as much as I had originally thought. Outside of the >>> paper, do we have the interface and semantics of these operations >>> described anywhere? I think that would help in trying to keep pace >>> with the use cases. >>> >>> The spirit of the suggestion was as a way to separate what (I think) >>> we can agree on as a first step (FT-MPI-like model) from the >>> communicator reconstruction, which I see as a secondary step. If we >>> stop to write up what the FT-MPI-like model should look like in the >>> standard, then I think we can push forward on other fronts >>> (prototyping of step 1, standardization of step 1, application >>> implementations using step 1) while still trying to figure out how >>> communication reconstruction should be expressed in the standard >>> such >>> that it is usable in target applications. >>> >>> So my motion is that the group explicitly focus effort on writing a >>> document describing the FT-MPI-like model we consider as a >>> foundation. Do so in the MPI standard language, and present it to >>> the >>> MPI Forum for a straw vote in the next couple of meetings. From this >>> document we can continue evolving it to support more advanced >>> features, like communicator reconstruction. >>> >>> I am willing to put effort into making such a document. However, I >>> would like explicit support from the working group in pursing such >>> an >>> effort, and the help of anyone interested in helping write-up/define >>> this specification. >>> >>> So what do people think taking this first step? >>> >>> -- Josh >>> >>> >>> On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: >>> >>>> I don't understand what you mean by "We can continue to pursue >>>> communicator reconstruction interfaces though a virtualization >>>> later above MPI." To me it seems that such interfaces will >>>> effectively need to implement communicators on top of MPI in order >>>> be operational, which will take about as much effort as >>>> implementing them inside MPI. In particular, I don't see a way to >>>> recreate a communicator using the MPI interface without making >>>> collective calls. However, we're defining MPI_Rejoin (or whatever >>>> its called) to be a local operation. This means that we cannot use >>>> the MPI communicators interface and must instead implement our own >>>> communicators. >>>> >>>> The bottom line is that it does make sense to start implementing >>>> support for the FT-MPI model and evolve that to a more elaborate >>>> model. However, I don't think that working on the rest above MPI >>>> will save us any effort or time. >>>> >>>> Greg Bronevetsky >>>> Post-Doctoral Researcher >>>> 1028 Building 451 >>>> Lawrence Livermore National Lab >>>> (925) 424-5756 >>>> bronevetsky1(a)llnl.gov >>>> >>>> At 01:17 PM 2/11/2009, Josh Hursey wrote: >>>>> In our meeting yesterday, I was sitting in the back trying to take >>>>> in >>>>> the complexity of communicator recreation. It seems that much of >>>>> the >>>>> confusion at the moment is that we (at least I) are still not >>>>> exactly >>>>> sure how the interface should be defined and implemented. >>>>> >>>>> I think of the process fault tolerance specification as a series >>>>> of >>>>> steps that can be individually specified building upon each step >>>>> while >>>>> working towards a specific goal set. From this I was asking >>>>> myself, is >>>>> there any foundational concepts that we can define now so that >>>>> folks >>>>> can start implementation. >>>>> >>>>> That being said I suggest that we consider FT-MPI's model of all >>>>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>>>> are >>>>> destroyed on a failure as the starting point for implementation. >>>>> This >>>>> would get us started. We can continue to pursue communicator >>>>> reconstruction interfaces though a virtualization later above MPI. >>>>> We >>>>> can use this layer to experiment with the communicator recreation >>>>> mechanisms in conjunction with applications while pursing the >>>>> first >>>>> step implementation. Once we start to agree on the interface for >>>>> communicator reconstruction, then we can start to push it into the >>>>> MPI >>>>> standard/library for a better standard/implementation. >>>>> >>>>> The communicator virtualization library is a staging area for >>>>> these >>>>> interface ideas that we seem to be struggling with. The >>>>> virtualization >>> >>> ------Original Message Truncated------ >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Greg Bronevetsky 12 Feb '09

12 Feb '09

>Heh OK. :) Oh well, maybe my statements can then provide some >motivation for users that are interested in FT at very least :). Right, and thanks for the comments! BTW, we are specifically worrying about limiting the effect of the API extensions on performance. However, I'm not sure how we'll deal with people who want to turn FT off. We tossed around the idea of multiple levels of FT support but I'm not sure where we left it. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

[Mpi3-ft] Fault recovery of multiple communication libraries
by Krishnamoorthy, Sriram 12 Feb '09

12 Feb '09

I would like to understand how MPI can be notified of a failure. Consider another communication library (ARMCI/GASnet/...) identifying a failure through its own mechanisms. In the current model, how can it notify MPI to verify/reconfigure/recover from the error, for example by performing an MPI communication to the failed process? Conversely, can a communication library register to be notified of an error that MPI identifies and recovers from, so that that the library can take appropriate action? Sriram.K -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of mpi3-ft-request(a)lists.mpi-forum.org Sent: Thursday, February 12, 2009 11:17 AM To: mpi3-ft(a)lists.mpi-forum.org Subject: mpi3-ft Digest, Vol 13, Issue 4 Send mpi3-ft mailing list submissions to mpi3-ft(a)lists.mpi-forum.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft or, via email, send a message with subject or body 'help' to mpi3-ft-request(a)lists.mpi-forum.org You can reach the person managing the list at mpi3-ft-owner(a)lists.mpi-forum.org When replying, please edit your Subject line so it is more specific than "Re: Contents of mpi3-ft digest..." Today's Topics: 1. Re: Communicator Virtualization as a step forward (Josh Hursey) 2. Re: Communicator Virtualization as a step forward (Graham, Richard L.) 3. Re: Communicator Virtualization as a step forward (George Bosilca) 4. Re: Communicator Virtualization as a step forward (Nathan DeBardeleben) ---------------------------------------------------------------------- Message: 1 Date: Thu, 12 Feb 2009 12:23:49 -0500 From: Josh Hursey <jjhursey(a)open-mpi.org> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Message-ID: <4554DA1F-BEF7-4F03-8B1C-5B5BF2783477(a)open-mpi.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Yeah I was planning on using that document as a starting point. I wanted to look over it again and see if anything needs to change given some of the discussions that we have been having in the group. It may also need some additional language about MPI2 interfaces. It has been while since I have looked over this particular document. I would also like to add some more control for process startup, but we may decide to take that on as a secondary step. Does UTK still have the LaTeX for that document somewhere? Do you know if Graham would be interested in participating in this development? Cheers, Josh On Feb 12, 2009, at 10:43 AM, George Bosilca wrote: > Josh, the document that you talk about already exist. It was published > in ISC'04. Here is the > link:http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT-M > PI.pdf > > george. ------------------------------ Message: 2 Date: Thu, 12 Feb 2009 14:01:15 -0500 From: "Graham, Richard L." <rlgraham(a)ornl.gov> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward To: mpi3-ft(a)lists.mpi-forum.org Message-ID: <537C6C0940C6C143AA46A88946B854170F0FC367(a)ORNLEXCHANGE.ornl.gov> Content-Type: text/plain; charset=UTF-8 Josh, Very early on in the process we got feedback from users that an ft-mpi like interface was of no interest to them. They would just as soon terminate the application and restart rather than use this sort of approach. Having said that, there is already previous demonstration that the ft-mpi approach is useful for some applications. If you look closely at the spec, the ft-mpi approach is a subset. of the current subset. I am working on pulling out the api's and expanding the explanations. The goal is to have this out before the next telecon in two weeks. Prototyping is under way, with ut, cray, and ornl committed to working on this. Right now supporting infrastructure is being developed. Your point on the mpi 2 interfaces is good. A couple of people had started to look at this when it looked like this might make it into the 2.2 version. The changes seemed to be more extensive than expected, so work stopped. This does need to be picked up on. Rich ------Original Message------ From: Josh Hursey To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Sent: Feb 12, 2009 8:31 AM Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward It is a good point that local communicator reconstruction operations require a fundamental change in the way communicators are handled by MPI. With that in mind it would probably take as much effort (if not more) to implement a virtualized version on top of MPI. So maybe it will not help as much as I had originally thought. Outside of the paper, do we have the interface and semantics of these operations described anywhere? I think that would help in trying to keep pace with the use cases. The spirit of the suggestion was as a way to separate what (I think) we can agree on as a first step (FT-MPI-like model) from the communicator reconstruction, which I see as a secondary step. If we stop to write up what the FT-MPI-like model should look like in the standard, then I think we can push forward on other fronts (prototyping of step 1, standardization of step 1, application implementations using step 1) while still trying to figure out how communication reconstruction should be expressed in the standard such that it is usable in target applications. So my motion is that the group explicitly focus effort on writing a document describing the FT-MPI-like model we consider as a foundation. Do so in the MPI standard language, and present it to the MPI Forum for a straw vote in the next couple of meetings. From this document we can continue evolving it to support more advanced features, like communicator reconstruction. I am willing to put effort into making such a document. However, I would like explicit support from the working group in pursing such an effort, and the help of anyone interested in helping write-up/define this specification. So what do people think taking this first step? -- Josh On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: > I don't understand what you mean by "We can continue to pursue > communicator reconstruction interfaces though a virtualization later > above MPI." To me it seems that such interfaces will effectively need > to implement communicators on top of MPI in order be operational, > which will take about as much effort as implementing them inside MPI. > In particular, I don't see a way to recreate a communicator using the > MPI interface without making collective calls. However, we're defining > MPI_Rejoin (or whatever its called) to be a local operation. This > means that we cannot use the MPI communicators interface and must > instead implement our own communicators. > > The bottom line is that it does make sense to start implementing > support for the FT-MPI model and evolve that to a more elaborate > model. However, I don't think that working on the rest above MPI will > save us any effort or time. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > At 01:17 PM 2/11/2009, Josh Hursey wrote: >> In our meeting yesterday, I was sitting in the back trying to take in >> the complexity of communicator recreation. It seems that much of the >> confusion at the moment is that we (at least I) are still not exactly >> sure how the interface should be defined and implemented. >> >> I think of the process fault tolerance specification as a series of >> steps that can be individually specified building upon each step >> while working towards a specific goal set. From this I was asking >> myself, is there any foundational concepts that we can define now so >> that folks can start implementation. >> >> That being said I suggest that we consider FT-MPI's model of all >> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >> are destroyed on a failure as the starting point for implementation. >> This would get us started. We can continue to pursue communicator >> reconstruction interfaces though a virtualization later above MPI. We >> can use this layer to experiment with the communicator recreation >> mechanisms in conjunction with applications while pursing the first >> step implementation. Once we start to agree on the interface for >> communicator reconstruction, then we can start to push it into the >> MPI standard/library for a better standard/implementation. >> >> The communicator virtualization library is a staging area for these >> interface ideas that we seem to be struggling with. The >> virtualization ------Original Message Truncated------ ------------------------------ Message: 3 Date: Thu, 12 Feb 2009 14:16:09 -0500 From: George Bosilca <bosilca(a)eecs.utk.edu> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Message-ID: <CD306C06-5662-4060-9E95-255852FE7BB1(a)eecs.utk.edu> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes I don't necessarily agree with the statement that FT-MPI is a subset of the current spec. As the current spec can be implemented on top of FT-MPI (with help from the PMPI interface), this tend to prove the opposite. However, I agree there are several features in the current spec that were not covered by the FT-MPI spec, but these features can be implemented on top of FT-MPI. As far as I understood, this is what Josh proposed, as this will give a quick start (i.e. FT-MPI implementation is already available). george. On Feb 12, 2009, at 14:01 , Graham, Richard L. wrote: > Josh, > Very early on in the process we got feedback from users that an ft- > mpi like interface was of no interest to them. They would just as > soon terminate the application and restart rather than use this sort > of approach. Having said that, there is already previous > demonstration that the ft-mpi approach is useful for some > applications. If you look closely at the spec, the ft-mpi approach is > a subset. of the current subset. > I am working on pulling out the api's and expanding the explanations. > The goal is to have this out before the next telecon in two weeks. > Prototyping is under way, with ut, cray, and ornl committed to > working on this. Right now supporting infrastructure is being > developed. > Your point on the mpi 2 interfaces is good. A couple of people had > started to look at this when it looked like this might make it into > the 2.2 version. The changes seemed to be more extensive than > expected, so work stopped. This does need to be picked up on. > > Rich > ------Original Message------ > From: Josh Hursey > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group > Sent: Feb 12, 2009 8:31 AM > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > It is a good point that local communicator reconstruction operations > require a fundamental change in the way communicators are handled by > MPI. With that in mind it would probably take as much effort (if not > more) to implement a virtualized version on top of MPI. So maybe it > will not help as much as I had originally thought. Outside of the > paper, do we have the interface and semantics of these operations > described anywhere? I think that would help in trying to keep pace > with the use cases. > > The spirit of the suggestion was as a way to separate what (I think) > we can agree on as a first step (FT-MPI-like model) from the > communicator reconstruction, which I see as a secondary step. If we > stop to write up what the FT-MPI-like model should look like in the > standard, then I think we can push forward on other fronts > (prototyping of step 1, standardization of step 1, application > implementations using step 1) while still trying to figure out how > communication reconstruction should be expressed in the standard such > that it is usable in target applications. > > So my motion is that the group explicitly focus effort on writing a > document describing the FT-MPI-like model we consider as a foundation. > Do so in the MPI standard language, and present it to the MPI Forum > for a straw vote in the next couple of meetings. From this document we > can continue evolving it to support more advanced features, like > communicator reconstruction. > > I am willing to put effort into making such a document. However, I > would like explicit support from the working group in pursing such an > effort, and the help of anyone interested in helping write-up/define > this specification. > > So what do people think taking this first step? > > -- Josh > > > On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: > >> I don't understand what you mean by "We can continue to pursue >> communicator reconstruction interfaces though a virtualization later >> above MPI." To me it seems that such interfaces will effectively >> need to implement communicators on top of MPI in order be >> operational, which will take about as much effort as implementing >> them inside MPI. In particular, I don't see a way to recreate a >> communicator using the MPI interface without making collective calls. >> However, we're defining MPI_Rejoin (or whatever its called) to be a >> local operation. This means that we cannot use the MPI communicators >> interface and must instead implement our own communicators. >> >> The bottom line is that it does make sense to start implementing >> support for the FT-MPI model and evolve that to a more elaborate >> model. However, I don't think that working on the rest above MPI will >> save us any effort or time. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> >> At 01:17 PM 2/11/2009, Josh Hursey wrote: >>> In our meeting yesterday, I was sitting in the back trying to take >>> in the complexity of communicator recreation. It seems that much of >>> the confusion at the moment is that we (at least I) are still not >>> exactly sure how the interface should be defined and implemented. >>> >>> I think of the process fault tolerance specification as a series of >>> steps that can be individually specified building upon each step >>> while working towards a specific goal set. From this I was asking >>> myself, is there any foundational concepts that we can define now so >>> that folks can start implementation. >>> >>> That being said I suggest that we consider FT-MPI's model of all >>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>> are destroyed on a failure as the starting point for implementation. >>> This >>> would get us started. We can continue to pursue communicator >>> reconstruction interfaces though a virtualization later above MPI. >>> We >>> can use this layer to experiment with the communicator recreation >>> mechanisms in conjunction with applications while pursing the first >>> step implementation. Once we start to agree on the interface for >>> communicator reconstruction, then we can start to push it into the >>> MPI standard/library for a better standard/implementation. >>> >>> The communicator virtualization library is a staging area for these >>> interface ideas that we seem to be struggling with. The >>> virtualization > > ------Original Message Truncated------ > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft ------------------------------ Message: 4 Date: Thu, 12 Feb 2009 12:16:23 -0700 From: Nathan DeBardeleben <ndebard(a)lanl.gov> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Message-ID: <49947587.6070604(a)lanl.gov> Content-Type: text/plain; charset=ISO-8859-1; format=flowed I really worry about taking the advise of users saying they would rather terminate and restart an application than having some assistance to help them ride through a problem. If they are worried about programming language/model changes, I would encourage them to open their eyes. Major programming model changes are predicted for > petascale computers and even petascale computers are having a hard time with classical MPI programming. I think we're more likely to see MPI as an underpinning of next-gen models. These users polled might not be extreme-scale users, however. Working at a laboratory positioning itself for exascale, we are intimately aware of the fact that "oh just rerun it" is a worthless conclusion. I wish I had more time to assist in this matter but our laboratory has cracked down on participation in things that are not directly associated with charge codes so it's a bit hard for me to spend any sizable amount of time. Please though, consider the user base when they say things like that. I'm sure Rich is well aware of these similar concerns. While MPI fault tolerance might not be important to the users running 1000 node systems, those of us approaching system mean time to interrupt under an hour are quite on the opposite side of that spectrum. Are the small-system users pushing for FT to not be inside of MPI? This is why I was so in favor of some sort of componentized MPI where users could exclude FT if they weren't worried about reliability (and thereby gain performance) but those of us who were in more dangerous reliability regimes could take the performance penalty and compile in / load in / configure in / whatever FT. -- Nathan --------------------------------------------------------------------- Nathan DeBardeleben, Ph.D. Los Alamos National Laboratory High Performance Computing Systems Integration (HPC-5) phone: 505-667-3428 email: ndebard(a)lanl.gov --------------------------------------------------------------------- Graham, Richard L. wrote: > Josh, > Very early on in the process we got feedback from users that an ft-mpi like interface was of no interest to them. They would just as soon terminate the application and restart rather than use this sort of approach. Having said that, there is already previous demonstration that the ft-mpi approach is useful for some applications. If you look closely at the spec, the ft-mpi approach is a subset. of the current subset. > I am working on pulling out the api's and expanding the explanations. The goal is to have this out before the next telecon in two weeks. > Prototyping is under way, with ut, cray, and ornl committed to working on this. Right now supporting infrastructure is being developed. > Your point on the mpi 2 interfaces is good. A couple of people had started to look at this when it looked like this might make it into the 2.2 version. The changes seemed to be more extensive than expected, so work stopped. This does need to be picked up on. > > Rich > ------Original Message------ > From: Josh Hursey > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > ReplyTo: MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group > Sent: Feb 12, 2009 8:31 AM > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > It is a good point that local communicator reconstruction operations > require a fundamental change in the way communicators are handled by > MPI. With that in mind it would probably take as much effort (if not > more) to implement a virtualized version on top of MPI. So maybe it > will not help as much as I had originally thought. Outside of the > paper, do we have the interface and semantics of these operations > described anywhere? I think that would help in trying to keep pace > with the use cases. > > The spirit of the suggestion was as a way to separate what (I think) > we can agree on as a first step (FT-MPI-like model) from the > communicator reconstruction, which I see as a secondary step. If we > stop to write up what the FT-MPI-like model should look like in the > standard, then I think we can push forward on other fronts > (prototyping of step 1, standardization of step 1, application > implementations using step 1) while still trying to figure out how > communication reconstruction should be expressed in the standard such > that it is usable in target applications. > > So my motion is that the group explicitly focus effort on writing a > document describing the FT-MPI-like model we consider as a foundation. > Do so in the MPI standard language, and present it to the MPI Forum > for a straw vote in the next couple of meetings. From this document we > can continue evolving it to support more advanced features, like > communicator reconstruction. > > I am willing to put effort into making such a document. However, I > would like explicit support from the working group in pursing such an > effort, and the help of anyone interested in helping write-up/define > this specification. > > So what do people think taking this first step? > > -- Josh > > > On Feb 11, 2009, at 5:57 PM, Greg Bronevetsky wrote: > > >> I don't understand what you mean by "We can continue to pursue >> communicator reconstruction interfaces though a virtualization later >> above MPI." To me it seems that such interfaces will effectively >> need to implement communicators on top of MPI in order be >> operational, which will take about as much effort as implementing >> them inside MPI. In particular, I don't see a way to recreate a >> communicator using the MPI interface without making collective calls. >> However, we're defining MPI_Rejoin (or whatever its called) to be a >> local operation. This means that we cannot use the MPI communicators >> interface and must instead implement our own communicators. >> >> The bottom line is that it does make sense to start implementing >> support for the FT-MPI model and evolve that to a more elaborate >> model. However, I don't think that working on the rest above MPI will >> save us any effort or time. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> >> At 01:17 PM 2/11/2009, Josh Hursey wrote: >> >>> In our meeting yesterday, I was sitting in the back trying to take >>> in the complexity of communicator recreation. It seems that much of >>> the confusion at the moment is that we (at least I) are still not >>> exactly sure how the interface should be defined and implemented. >>> >>> I think of the process fault tolerance specification as a series of >>> steps that can be individually specified building upon each step >>> while working towards a specific goal set. From this I was asking >>> myself, is there any foundational concepts that we can define now so >>> that folks can start implementation. >>> >>> That being said I suggest that we consider FT-MPI's model of all >>> communicators except the base 3 (COMM_WORLD, COMM_SELF, COMM_NULL) >>> are destroyed on a failure as the starting point for implementation. >>> This would get us started. We can continue to pursue communicator >>> reconstruction interfaces though a virtualization later above MPI. >>> We can use this layer to experiment with the communicator recreation >>> mechanisms in conjunction with applications while pursing the first >>> step implementation. Once we start to agree on the interface for >>> communicator reconstruction, then we can start to push it into the >>> MPI standard/library for a better standard/implementation. >>> >>> The communicator virtualization library is a staging area for these >>> interface ideas that we seem to be struggling with. The >>> virtualization >>> > > ------Original Message Truncated------ > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------ _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft End of mpi3-ft Digest, Vol 13, Issue 4 **************************************

1 0