February 2009 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Greg Bronevetsky 19 Feb '09

19 Feb '09

I understand why you want to specify what a level of support means but I think that this cannot and should not be done as part of the MPI spec. It cannot be done because we simply have no concepts in MPI such as a switch or a router or a compute node. As such, how do we talk about the failure of one of these? Also, I cannot imagine that we'll be able to foresee the failure modes of systems that will be fielded in 10 years. How can we possibly write about them? We can draft a separate document of best practices that provides specific examples on specific systems. However, such details don't belong in the spec. It should not be done because it is unnecessary. Users don't care about which events trigger fault notification. They care that the rate of unrecoverable errors is lower than a given bound and the rate or recoverable errors is lower than some other, much looser bound. A given system (MPI + hardware) can satisfy this requirement in such a wide variety of ways that it is simply inadvisable to restrict it. For example, the system designers can simply spend more money on more reliable hardware. This is the choice made by BlueGene. Others may save money on hardware but provide a more reliable MPI that survives the failures that happen to be most common on this system. Given that it is a bad idea to put these constraints in the MPI spec, I suggest two options that we can take. One is to write a separate document documenting some best practices. Another is to be very public about the techniques used to make the reference implementation support fault notification so that other implementations can follow suit. In the end I expect this feature to follow system designers and user demands. If a given user writes into their contract a certain bound on unrecoverable failures, the system vendor will have to work out how much money/performance to spend on hardware and how much on improved fault notification support. The entire point of the fault notification API is to make it possible for users and system designers to talk about recoverable vs unrecoverable failures, rather than just unrecoverable ones. In the end, I think that the whole thing will be driven by the reliability of large-scale companies such as US national labs. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov http://greg.bronevetsky.com At 07:18 AM 2/19/2009, Supalov, Alexander wrote: >Thanks, this makes sense. > >I expect that performance overheads will be noticeable, so that >there will most likely be FT- and nonFT-versions of the MPI >libraries, just like now there are MT- and nonMT-versions, basically >for the same reason. Still, even there people can ask for different >level of MT support, thus matching their actual needs to the level >of service provided and thus the expected overheads. > >This is where I want to cite your reply: "since most MPI >implementations want to support large-scale systems as well as >smaller ones, these implementations will provide a way for >applications to request different levels of fault notification support". > >If this is foreseeable, why not helping out with this right now? Do >we think the problem is too hard to solve or do we want to let MPIs >settle into their ways and thus practically identify the most >reasonable levels to be standardized later? > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg Bronevetsky >Sent: Tuesday, February 17, 2009 6:18 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > >Yes, this is definitely a topic for voice conversation but let me >continue this over email because I think that I can describe my >position more precisely based on your comments. The definition of >whether fault notification API is working or not is Not whether the >underlying MPI implementation is guaranteed to do something when some >low-level event happens. It is working if the rate of unrecoverable >failures is below on application-specified bound, with all the >remaining failures converted into notifications. This is the same for >MPI_Send/MPI_Recv, which work not if they provide a certain level of >performance but whether they deliver data or not. These are both >high-level specifications and we don't care about how they are >implemented at the network level. > >Having said that, there is one large difference between the two >cases, which is the social dynamics. The MPI_Send/MPI_Recv API has >non-trivial support in all MPI implementations and you can get better >MPI performance by either using the same MPI implementation on a >better machine or by switching to a higher performing MPI >implementation. This fluidity works to help encourage MPI >implementations to support MPI_Send/MPI_Recv because application >developers will be confident that they'll get good use out them. > >In contrast, with the fault notification API the odds are that some >MPI implementations will provide trivial support for the fault >notification API while others will provide good or configurable >support. As such, if a developer wants their unrecoverable error rate >to be below a certain bound, they will have less freedom to match MPI >implementations to systems because if a given machine has too high an >error rate, an unreliable MPI implementation will not provide >sufficient reliability no matter what you do. You will need to switch >to a different MPI implementation. Your claim is that this will cause >application developers to not bother coding fault tolerance into >their applications because they can't be sure that it will do any >good on their favorite MPI implementation. This makes sense but I >believe that it is incorrect. > >The reason is that the driver for fault tolerance is not applications >or MPI implementations but the hardware systems. Today's large >machines are sufficiently unreliable that today's applications have >to provide their own fault tolerance solutions without any support >from MPI. For large-scale applications, fault tolerance is not a >luxury but a necessity, so if they need to switch MPIs in order to >get it, they will (assuming that not too much performance is lost). >If an existing application only runs on small, reliable systems, it >will never need to be recoded to use the fault notification API. >However, if it grows to a large enough scale for faults to be a >problem, fault notification support will need to be coded in because >there is no other choice besides just failing all the time. As such, >the reduced "liquidity" of the fault notification API is more than >offset by the strong driving force that system failures have on >large-scale applications. Using it will not be a choice but a >necessity and as such, there will be a sub-community of >applications/MPI implementations that use it/provide it, and the rest >who ignore it. More likely, since most MPI implementations want to >support large-scale systems as well as smaller ones, these >implementations will provide a way for applications to request >different levels of fault notification support, which will then >overcome the "liquidity" problem that you've identified. > >So that's my argument in favor of fault notification. Fault >notification is a well-defined and useful API that is less "liquid" >than much of the existing MPI specification. However, because there >is a significant sub-community of applications and systems on which >faults are a real problem, this weakness will be more than offset by >the sheer necessity of running on systems that fail frequently. In >this context the fault notification API will allow MPI implementation >to bound the rate of unrecoverable failures even on unreliable >hardware platforms. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov >http:// greg.bronevetsky.com > >At 04:55 AM 2/17/2009, Supalov, Alexander wrote: > >Hi, > > > >Thanks. I think we're coming from different corners, and the main > >problem is that the two criteria of yours, namely, "self-consistent > >specification and usefulness of the chosen abstraction level" are > >not sufficient for me. The key here appears to be in the > >"usefulness" that you seem to understand as "this can be used in > >principle on my future Petascale machine running my favorite MPI" > >and mine "to be used in a real application, an API should be useful > >in a wide, a priori known variety of practically relevant situations > >that do not depend on the MPI implementation at hand or the > platform involved". > > > >Maybe we should have a call or something to discuss this live, > >without email getting into way. The rest is just an outline of what > >I'd have to say if we met. > > > >I say that we should define a standard set of the faults that will > >be detected, and then say what implementations should provide what > >level of service to be compliant with a particular level of the FT > >support that the standard is to specify in clear terms. If we don't > >do this, we will only prepare a good ground for MPI-4, were we will > >have to fix the flaws of the MPI-3. I'm afraid that at the moment > >we're driving in this very direction, and this is why. > > > >The ultimate test of a spec is a number of implementations that are > >widely used as specified. I'm concerned that the current > >notification spec is too weak to be appealing to the commercial > >users and implementors. I'm concerned about this because the > >semantics of an API that kicks in under unknown circumstances are > >surely ill-defined, and the ROI of using it in any given application > >cannot be assessed upfront. This means that the investment is > >unlikely to be made - either on the commercial MPI implementation or > >on the commercial MPI application side of the equation. > > > >In other words, why would I care to use an API if I were not aware > >of when and how this would help my application run weeks rather than > >days between the faults? And I don't mean here a student with a > >diploma or PhD thesis to write and forget about. I mean real life > >commercial developers who need to justify every bit of what they are > >doing by showing positive ROI to their management in the times of > >worldwide economic crisis. Justify by promising to earn more money > >than they are going to spend for the development, or else. > > > >Now, I can't comment on the networking topology example of yours > >because I cannot fully follow the logic. Let me try to give another, > >hopefully more practical example. > > > >Consider interaction of the MPI_Init/MPI_Finalize with the > >underlying OS and job managers. It's about as undefined as that of > >the mentioned calls with the checkpointer. Nevertheless, many usable > >implementations can cope with that quite nicely. By the way, this is > >how the checkpoint/restarting could work as well, providing the > >checkpointer at hand with the MPI ready to be checkpointed/restarted > >at a well defined point of the MPI program. After all, this is how > >it is done now: a configuration flag tells the MPI what checkpointer > >to expect. This could be refined, turned into dynamic recognition of > >the active checkpointer, etc. This is all trivial. > > > >Imagine now that we have an API instead of the MPI_Init/MPI_Finalize > >that says: "Well, it sort of starts and terminates an MPI job, but > >one cannot guarantee how many processes will be started nor whether > >they are usable after this because one cannot explain really in what > >conditions the above is true. Anyway, this is all too low level for > >the MPI standard to deal with. So be it because we say this > >interface is self-contained and useful." > > > >The provided interface is self-consistent. It sort of starts a job. > >Is it practically useful? No, because one cannot know when it works > >at all. What is the expected commercial user reaction to this > >interface? Steer clear of it or, even worse, assume on the limited > >past student experience that the MPI implementation at hand is the > >only correct one, and then blame all others that they do a sloppy > >job when the product should be ported elsewhere. Neither would be > >good for the MPI standard. > > > >Cf. process spawning and 1-sided comm, the biggest failures of the > >MPI standard to date in practical terms. Were they self-consistent > >and useful in the sense that you advocate? Sure. Were they useful in > >the sense that I advocate? No. And they duly faded into obscurity, > >because they did not pass the ultimate test even though they did > >meet the criteria that you claim are necessary and sufficient for > >the MPI standard. I pray you consider this before we exchange > >another couple of emails. I'm just trying to help. > > > >Best regards. > > > >Alexander > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > >Sent: Monday, February 16, 2009 5:58 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > >MPI never places conditions on how the MPI implementation does its > >job. It never says whether MPI uses static or dynamic routing or how > >much performance degrades as a result of the application using a > >particular communication pattern, which depends closely on the > >physical network topology. These are simply issues that are too low > >level for the MPI spec to define although they very much matter to > >application developers. What it does instead is define a set of > >semantics for MPI_Send, MPI_Recv, etc. that apply regardless of all > >those details and accepts the compromise: its an internally > >self-consistent spec that doesn't fully define all the relevant facts > >about the system. They key thing is that it is a useful compromise. > >Thus, we have two tests for a candidate API: self-consistent > >specification and usefulness of the chosen abstraction level. > > > >The fault notification API makes exactly the same compromise as the > >overall MPI spec. It doesn't say anything about how faults are > >detected since that is a very low-level network matter. However, it > >presents a self-consistent high-level specification that allows > >applications to react to any such errors. Furthermore, it is clearly > >useful. It is a red herring to worry about which low-level events > >will cause which high-level notifications. The only relevant thing is > >the probability of unrecoverable errors. Applications do not want > >their applications to randomly abort with a frequency higher than > >once every few days or weeks. If it is higher, those unrecoverable > >failures must be converted into recoverable failures by the MPI > >library and given to the application via the fault notification API. > >This is the entire function of the fault notification API: to allow > >MPI to convert unrecoverable system failures (currently they're all > >unrecoverable) into recoverable failures. This makes it possible for > >customers to buy systems that fail relatively frequently while making > >them usable by making their applications fault tolerant. Thus, the > >fault notification API is both self-consistent and useful, passing > >both tests of the MPI spec. > > > >In contrast, the checkpointing API is useful but not self-consistent > >API. Its semantics require details (i.e. interactions with the > >checkpointer) that are too low-level to be specified in the MPI spec. > >As a result, it needs additional mechanisms that allow individual MPI > >implementations to provide the information that cannot be detailed in > >the MPI spec. > > > >Thus, these two APIs are not at all similar unless you wish to argue > >that 1. the MPI spec is ill-defined because it doesn't specify the > >network topology or that 2. the semantics of being notified of a > >fault are ill-defined. If you wish to argue the latter, I would love > >to see examples because they would need to be fixed before this API > >is ready to go before the forum. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > > > >At 06:04 AM 2/16/2009, Supalov, Alexander wrote: > > >Thanks. I think that since the notification API does not provide any > > >guarantee as to what kind of faults is treated how, the whole thing > > >becomes a negotiation between the MPI implementation and the > > >underlying networking layers. Moreover, it becomes a negotiation of > > >sorts between the application and the MPI implementation, because > > >the application cannot know upfront what faults will be treated what way. > > > > > >This is, in my mind, is very comparable to, if not worse than the > > >negotiation between the MPI_prepare_for_checkpoint & > > >MPI_Restart_after_chekpoint implementation on one hand, and the > > >checkpointer involved on the other hand. > > > > > >Frankly, I don't see any difference here, or, if any, one in favor > > >of the checkpointing interface. > > > > > >Anyway, thanks for clarification. > > > > > >-----Original Message----- > > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > > >Sent: Friday, February 13, 2009 7:18 PM > > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > > > >At 10:10 AM 2/13/2009, Supalov, Alexander wrote: > > > >Thanks. Could you please clarify to me, if possible, using some > > > >practically relevant example, how fault notification for a set of > > > >undefined fault types that may vary from MPI implementation to > > > >implementation differs from the equally abstract > > > >MPI_Checkpoint/MPI_Restart that semantically clearly prepare the MPI > > > >implementation at hand for the checkpoint action done by the > > > >checkpointing system involved, and then semantically clearly recover > > > >the MPI part of the program after the system restore? > > > > > >Simple. As you've pointed out, the checkpointing API is well defined > > >from the application's point of view. However, its semantics are weak > > >from the checkpointer's point of view. Seen from this angle, it is > > >not clear what the checkpointer can expect from the MPI library and > > >the whole thing devolves into a negotiation between individual > > >checkpointers and individual MPI libraries on a variety of specific > > >system configurations. In contrast, the fault notification API only > > >has an application view, which is in fact well-defined. The weakness > > >of the fault notification API is what you've already described, that > > >it provides no guarantees about the quality of the implementation in > > >a way that is more significant than for other portions of MPI, such > > >as network details for MPI_Send/MPI_Recv. > > > > > >Greg Bronevetsky > > >Post-Doctoral Researcher > > >1028 Building 451 > > >Lawrence Livermore National Lab > > >(925) 424-5756 > > >bronevetsky1(a)llnl.gov > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >--------------------------------------------------------------------- > > >Intel GmbH > > >Dornacher Strasse 1 > > >85622 Feldkirchen/Muenchen Germany > > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > > >VAT Registration No.: DE129385895 > > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > > > >This e-mail and any attachments may contain confidential material for > > >the sole use of the intended recipient(s). Any review or distribution > > >by others is strictly prohibited. If you are not the intended > > >recipient, please contact the sender and delete all copies. > > > > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Supalov, Alexander 19 Feb '09

19 Feb '09

Thanks, this makes sense. I expect that performance overheads will be noticeable, so that there will most likely be FT- and nonFT-versions of the MPI libraries, just like now there are MT- and nonMT-versions, basically for the same reason. Still, even there people can ask for different level of MT support, thus matching their actual needs to the level of service provided and thus the expected overheads. This is where I want to cite your reply: "since most MPI implementations want to support large-scale systems as well as smaller ones, these implementations will provide a way for applications to request different levels of fault notification support". If this is foreseeable, why not helping out with this right now? Do we think the problem is too hard to solve or do we want to let MPIs settle into their ways and thus practically identify the most reasonable levels to be standardized later? -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Tuesday, February 17, 2009 6:18 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward Yes, this is definitely a topic for voice conversation but let me continue this over email because I think that I can describe my position more precisely based on your comments. The definition of whether fault notification API is working or not is Not whether the underlying MPI implementation is guaranteed to do something when some low-level event happens. It is working if the rate of unrecoverable failures is below on application-specified bound, with all the remaining failures converted into notifications. This is the same for MPI_Send/MPI_Recv, which work not if they provide a certain level of performance but whether they deliver data or not. These are both high-level specifications and we don't care about how they are implemented at the network level. Having said that, there is one large difference between the two cases, which is the social dynamics. The MPI_Send/MPI_Recv API has non-trivial support in all MPI implementations and you can get better MPI performance by either using the same MPI implementation on a better machine or by switching to a higher performing MPI implementation. This fluidity works to help encourage MPI implementations to support MPI_Send/MPI_Recv because application developers will be confident that they'll get good use out them. In contrast, with the fault notification API the odds are that some MPI implementations will provide trivial support for the fault notification API while others will provide good or configurable support. As such, if a developer wants their unrecoverable error rate to be below a certain bound, they will have less freedom to match MPI implementations to systems because if a given machine has too high an error rate, an unreliable MPI implementation will not provide sufficient reliability no matter what you do. You will need to switch to a different MPI implementation. Your claim is that this will cause application developers to not bother coding fault tolerance into their applications because they can't be sure that it will do any good on their favorite MPI implementation. This makes sense but I believe that it is incorrect. The reason is that the driver for fault tolerance is not applications or MPI implementations but the hardware systems. Today's large machines are sufficiently unreliable that today's applications have to provide their own fault tolerance solutions without any support from MPI. For large-scale applications, fault tolerance is not a luxury but a necessity, so if they need to switch MPIs in order to get it, they will (assuming that not too much performance is lost). If an existing application only runs on small, reliable systems, it will never need to be recoded to use the fault notification API. However, if it grows to a large enough scale for faults to be a problem, fault notification support will need to be coded in because there is no other choice besides just failing all the time. As such, the reduced "liquidity" of the fault notification API is more than offset by the strong driving force that system failures have on large-scale applications. Using it will not be a choice but a necessity and as such, there will be a sub-community of applications/MPI implementations that use it/provide it, and the rest who ignore it. More likely, since most MPI implementations want to support large-scale systems as well as smaller ones, these implementations will provide a way for applications to request different levels of fault notification support, which will then overcome the "liquidity" problem that you've identified. So that's my argument in favor of fault notification. Fault notification is a well-defined and useful API that is less "liquid" than much of the existing MPI specification. However, because there is a significant sub-community of applications and systems on which faults are a real problem, this weakness will be more than offset by the sheer necessity of running on systems that fail frequently. In this context the fault notification API will allow MPI implementation to bound the rate of unrecoverable failures even on unreliable hardware platforms. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov http://greg.bronevetsky.com At 04:55 AM 2/17/2009, Supalov, Alexander wrote: >Hi, > >Thanks. I think we're coming from different corners, and the main >problem is that the two criteria of yours, namely, "self-consistent >specification and usefulness of the chosen abstraction level" are >not sufficient for me. The key here appears to be in the >"usefulness" that you seem to understand as "this can be used in >principle on my future Petascale machine running my favorite MPI" >and mine "to be used in a real application, an API should be useful >in a wide, a priori known variety of practically relevant situations >that do not depend on the MPI implementation at hand or the platform involved". > >Maybe we should have a call or something to discuss this live, >without email getting into way. The rest is just an outline of what >I'd have to say if we met. > >I say that we should define a standard set of the faults that will >be detected, and then say what implementations should provide what >level of service to be compliant with a particular level of the FT >support that the standard is to specify in clear terms. If we don't >do this, we will only prepare a good ground for MPI-4, were we will >have to fix the flaws of the MPI-3. I'm afraid that at the moment >we're driving in this very direction, and this is why. > >The ultimate test of a spec is a number of implementations that are >widely used as specified. I'm concerned that the current >notification spec is too weak to be appealing to the commercial >users and implementors. I'm concerned about this because the >semantics of an API that kicks in under unknown circumstances are >surely ill-defined, and the ROI of using it in any given application >cannot be assessed upfront. This means that the investment is >unlikely to be made - either on the commercial MPI implementation or >on the commercial MPI application side of the equation. > >In other words, why would I care to use an API if I were not aware >of when and how this would help my application run weeks rather than >days between the faults? And I don't mean here a student with a >diploma or PhD thesis to write and forget about. I mean real life >commercial developers who need to justify every bit of what they are >doing by showing positive ROI to their management in the times of >worldwide economic crisis. Justify by promising to earn more money >than they are going to spend for the development, or else. > >Now, I can't comment on the networking topology example of yours >because I cannot fully follow the logic. Let me try to give another, >hopefully more practical example. > >Consider interaction of the MPI_Init/MPI_Finalize with the >underlying OS and job managers. It's about as undefined as that of >the mentioned calls with the checkpointer. Nevertheless, many usable >implementations can cope with that quite nicely. By the way, this is >how the checkpoint/restarting could work as well, providing the >checkpointer at hand with the MPI ready to be checkpointed/restarted >at a well defined point of the MPI program. After all, this is how >it is done now: a configuration flag tells the MPI what checkpointer >to expect. This could be refined, turned into dynamic recognition of >the active checkpointer, etc. This is all trivial. > >Imagine now that we have an API instead of the MPI_Init/MPI_Finalize >that says: "Well, it sort of starts and terminates an MPI job, but >one cannot guarantee how many processes will be started nor whether >they are usable after this because one cannot explain really in what >conditions the above is true. Anyway, this is all too low level for >the MPI standard to deal with. So be it because we say this >interface is self-contained and useful." > >The provided interface is self-consistent. It sort of starts a job. >Is it practically useful? No, because one cannot know when it works >at all. What is the expected commercial user reaction to this >interface? Steer clear of it or, even worse, assume on the limited >past student experience that the MPI implementation at hand is the >only correct one, and then blame all others that they do a sloppy >job when the product should be ported elsewhere. Neither would be >good for the MPI standard. > >Cf. process spawning and 1-sided comm, the biggest failures of the >MPI standard to date in practical terms. Were they self-consistent >and useful in the sense that you advocate? Sure. Were they useful in >the sense that I advocate? No. And they duly faded into obscurity, >because they did not pass the ultimate test even though they did >meet the criteria that you claim are necessary and sufficient for >the MPI standard. I pray you consider this before we exchange >another couple of emails. I'm just trying to help. > >Best regards. > >Alexander > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg Bronevetsky >Sent: Monday, February 16, 2009 5:58 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > >MPI never places conditions on how the MPI implementation does its >job. It never says whether MPI uses static or dynamic routing or how >much performance degrades as a result of the application using a >particular communication pattern, which depends closely on the >physical network topology. These are simply issues that are too low >level for the MPI spec to define although they very much matter to >application developers. What it does instead is define a set of >semantics for MPI_Send, MPI_Recv, etc. that apply regardless of all >those details and accepts the compromise: its an internally >self-consistent spec that doesn't fully define all the relevant facts >about the system. They key thing is that it is a useful compromise. >Thus, we have two tests for a candidate API: self-consistent >specification and usefulness of the chosen abstraction level. > >The fault notification API makes exactly the same compromise as the >overall MPI spec. It doesn't say anything about how faults are >detected since that is a very low-level network matter. However, it >presents a self-consistent high-level specification that allows >applications to react to any such errors. Furthermore, it is clearly >useful. It is a red herring to worry about which low-level events >will cause which high-level notifications. The only relevant thing is >the probability of unrecoverable errors. Applications do not want >their applications to randomly abort with a frequency higher than >once every few days or weeks. If it is higher, those unrecoverable >failures must be converted into recoverable failures by the MPI >library and given to the application via the fault notification API. >This is the entire function of the fault notification API: to allow >MPI to convert unrecoverable system failures (currently they're all >unrecoverable) into recoverable failures. This makes it possible for >customers to buy systems that fail relatively frequently while making >them usable by making their applications fault tolerant. Thus, the >fault notification API is both self-consistent and useful, passing >both tests of the MPI spec. > >In contrast, the checkpointing API is useful but not self-consistent >API. Its semantics require details (i.e. interactions with the >checkpointer) that are too low-level to be specified in the MPI spec. >As a result, it needs additional mechanisms that allow individual MPI >implementations to provide the information that cannot be detailed in >the MPI spec. > >Thus, these two APIs are not at all similar unless you wish to argue >that 1. the MPI spec is ill-defined because it doesn't specify the >network topology or that 2. the semantics of being notified of a >fault are ill-defined. If you wish to argue the latter, I would love >to see examples because they would need to be fixed before this API >is ready to go before the forum. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov > >At 06:04 AM 2/16/2009, Supalov, Alexander wrote: > >Thanks. I think that since the notification API does not provide any > >guarantee as to what kind of faults is treated how, the whole thing > >becomes a negotiation between the MPI implementation and the > >underlying networking layers. Moreover, it becomes a negotiation of > >sorts between the application and the MPI implementation, because > >the application cannot know upfront what faults will be treated what way. > > > >This is, in my mind, is very comparable to, if not worse than the > >negotiation between the MPI_prepare_for_checkpoint & > >MPI_Restart_after_chekpoint implementation on one hand, and the > >checkpointer involved on the other hand. > > > >Frankly, I don't see any difference here, or, if any, one in favor > >of the checkpointing interface. > > > >Anyway, thanks for clarification. > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > >Sent: Friday, February 13, 2009 7:18 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > >At 10:10 AM 2/13/2009, Supalov, Alexander wrote: > > >Thanks. Could you please clarify to me, if possible, using some > > >practically relevant example, how fault notification for a set of > > >undefined fault types that may vary from MPI implementation to > > >implementation differs from the equally abstract > > >MPI_Checkpoint/MPI_Restart that semantically clearly prepare the MPI > > >implementation at hand for the checkpoint action done by the > > >checkpointing system involved, and then semantically clearly recover > > >the MPI part of the program after the system restore? > > > >Simple. As you've pointed out, the checkpointing API is well defined > >from the application's point of view. However, its semantics are weak > >from the checkpointer's point of view. Seen from this angle, it is > >not clear what the checkpointer can expect from the MPI library and > >the whole thing devolves into a negotiation between individual > >checkpointers and individual MPI libraries on a variety of specific > >system configurations. In contrast, the fault notification API only > >has an application view, which is in fact well-defined. The weakness > >of the fault notification API is what you've already described, that > >it provides no guarantees about the quality of the implementation in > >a way that is more significant than for other portions of MPI, such > >as network details for MPI_Send/MPI_Recv. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Supalov, Alexander 19 Feb '09

19 Feb '09

Thanks. What would be the best API for this message logging approach? It looks a little bit like journal file system, doesn't it? -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Wednesday, February 18, 2009 8:10 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >First, this API works only for the synch-and-stop protocol. It does >not support any other protocol and this will lead into bad places in >the future because this protocol scales poorly with increasing >failure rates since it requires all processes to participate in every >single recovery. I can discuss the reasons for my assertion in more >detail if you'd like. > >AS> What other protocols are you talking about? I did not tell >anywhere whether these calls synch anything, or whether they are >collective, or whether the checkpointer is called on all nodes. This >remains to be defined. If you would please define other scenarios, >including those that scale well, I bet we'd probably be able to map >them to these calls that way or another. Ok, lets look as message logging. Each process checkpoints independently but logs all non-deterministic events and the data of sent or received messages (details and overheads depend on protocol). If a process checkpoints, we can erase all the logged non-determinism or message data that relates to events before this checkpoint. If some process fails, it rolls back to its last checkpoint and all of the other processes resend to 1. the data of all messages they sent to it since that checkpoint and 2. the outcomes of all the nonderministic events that it experienced since that checkpoint. I know how to implement this either inside MPI or above MPI with the aid for the fault notification API. These things already exist, for example in MPICH-V. I can also imagine an API that lets the application ask MPI to run this protocol on its behalf. However, this API will not require quiescence or stable states, or anything like that. In particular, quiescence is completely incompatible with message logging. Realistically speaking, this procotol can only be provided to the application without any direct interaction with the application. This is fine but it also suggests that we don't need much of an API to support it. The same applies to other protocols that need to communicate before, after and/or during the checkpoint. As for your other points, Thomas' email covers roughly the same points that I would have made. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov http://greg.bronevetsky.com >Second, this API is not compatible with application-level >checkpointing. An application-level checkpointer can deal with >internal MPI state in one of two ways. One option is to checkpoint it >at a high-level by using only the existing MPI calls, performing the >appropriate tracking of messages at calls to MPI_Send and MPI_Recv. >The simplest example of this is to ensure that there is no >communication going on at the time of the checkpoint, and then save >each process' state immediately after calling MPI_Barrier(). If this >is the option used, there is no need for any API extensions. > >AS> One option for the application level checkpointing is to save >the apps specific state (those two vectors) without saving the state >of the MPI at all. This will work if the MPI state is quiet (no >connections, nothing), and the application can be sure that its >state will be good, and when it restarts, the MPI hitting the >MPI_restore_after_checkpoint will know how to re-establish all >connections, etc. I think this is more than an MPI_Barrier, isn't it? > >The second option is for the application to somehow same MPI internal >data structures, which is what the MPI_Prepare_for_checkpoint call is >for: to make sure that all MPI state is in locations accessible by >the application. Where is that? Well, it depends on the operating >system and the hardware. Does the application need to checkpoint and >restore MPI's open file handles or shared memory regions or is MPI >supposed to let those go during MPI_Prepare_for_checkpoint? The point >is, all these details cannot be defined in the MPI spec and if this >is the case, the call will not be sufficiently well-defined to be >used by an application-level checkpointer. However, it can be used by >the application to initiate a system-level checkpoint since in this >case all those details will not be the application's problem. > >AS> As for the place where all relevant MPI data is to be stored at >this point in time, we should not care - the MPI implementation and >the checkpointer involved will agree on this when the MPI will be >configured to use this particular checkpointer, either statically or >dynamically. The checkpointer will know that the data is in the >"right" place when it gats control after the MPI_prepare_for_checkpoint call. > >Finally, it does support system-initiated checkpointing. If the >system wants to start a checkpoint in the middle of an MPI_Send() >call, this API provides no help. If we care about this use-case we >need to allow checkpoints to be taken at all times or provide a way >to register a callback that tells the system when a checkpoint can >next be taken. > >AS> I suppose this is "does not support", right? If so, let's simply >add this callback registration call, say, >MPI_REGISTER_CHECKPOINT_CALLBACK, that will define the call to be >made when the MPI_Prepare_for_checkpoint is about to return control >to the application. This will I guess address your concern, won't it? > >The bottom line given the above issues is that the only benefit this >API does have is providing a common set of names for functions that >have poor semantics and limited functionality. This benefit does not >pass the bar. > >AS> See above. I think you raise valid points, but all of them can >be adequately addressed given the desire to provide the >checkpoint-restart capability > >If I'm allowed to digress here and refer back to the fault >notification, note that the checkpoint/restart capability is >something that does guarantee fault tolerance, also in the case when >fatal unrecoverable errors occur. You "simply" fix the machine and >restart from the earlier checkpoint. > >As such, this widely used and proven capability may be a safe first >bet for the FT WG to introduce as something people will like and >will use. Much more likely, in fact, than the notification stuff >that yet needs to be properly defined, and as currently envisioned, >with all that uncertain communicator restoration stuff, will require >substantial application rethinking and repgrogramming to be used. > >After all, this is only a matter of priorities. Going for a low >hanging fruit like checkpoint/restart may be unromantic, but it's >definitely very pragmatic and sound. And in any case, I call upon >the same set of criteria to be used for all things that we consider. >Usability in the sense that a feature will be used by commercial >apps is definitely one of them. Or so I think. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov >http:// greg.bronevetsky.com > >At 09:27 AM 2/18/2009, Supalov, Alexander wrote: > >I see. Let me explain what I think those two calls would do to > >facilitate checkpointing. > > > >The program piece would look as follows: > > > >CALL MPI_PREPARE_FOR_CHECKPOINT > > > >! Whatever calls are necessary to do the actual checkpointing, > >including, in the case of resident monitoring checkpointer, none. > > > >CALL MPI_RESTORE_AFTER_CHECKPOINT > > > >The first call will bring MPI into a state that is acceptable to the > >checkpointer involved. This may include quiescence, etc. > > > >Then the checkpointer will do its job. Note that this may be > >application level checkpointer that saves two important arrays. Or > >this may be a system checkpointer that dumps the memory, etc. > >Finally, this can be a checkpointer that is notified when the > >MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in > >some way without the program calling it. > > > >The second call will restore the MPI state to what it was, > >functionally, after the checkpoint. "Functionally" here means that > >this may include running on a different set of nodes, etc. > > > >Is it this scenario that you think does not pass the bar? If so, > in what way? > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > >Sent: Wednesday, February 18, 2009 6:20 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > >My point is that each MPI implementation will have to negotiate with > >each checkpointer what this call will mean on each platform, meaning > >that it is upto the implementation-specific call to do all that > >magic. It doesn't save the application any real effort to call this > >routine by its implementation-specific name rather than a unified > >name given that the other two major problems are not solved: > >- Providing a uniform way for checkpointers and MPI implementations > >to interact (THE major for having this proposal in the first place) and > >- Support for more than the synch-and-stop checkpointing protocol, > >which does scales poorly with rising failure rates. > > > >My point here is that covering one of the three major issues is not > >good enough to modify the MPI specification and I don't think that > >we'll ever be able to do more than that. We're pretty much agreed in > >the working group that the first point above cannot be done and I > >don't think that the second can be done without providing callbacks > >to be called for each incoming/outgoing message. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > >http:// greg.bronevetsky.com > > > >At 08:40 AM 2/18/2009, Supalov, Alexander wrote: > > >Thanks. How will the checkpointer-specific call talk to the MPI? > > > > > >-----Original Message----- > > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > > >Sent: Wednesday, February 18, 2009 5:29 PM > > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > > > > > > > >Thanks. How will you let the MPI know the checkpoint is coming, to > > > >give it a fair chance to prepare to this and then recover after the > > > >checkpoint? This is akin to the MPI_Finalize/MPI_Init in some sense, > > > >midway thru the job, hence the analogy. > > > > > >Just use the checkpointer-specific call. The call is going to have > > >checkpointer-specific semantics, so why not give it a > > >checkpointer-specific name? I understand that there is some use to > > >allowing applications to use the same name across all checkpointers > > >but the bar should be higher than that for adding something to the > > >standard. Also, right now the whole approach inherently only supports > > >one checkpointing protocol: synch-and-stop. If we can work out a more > > >generic API that supports other protocols I think that it may have > > >enough value to be included in the spec. Right now it still hasn't > > >passed the bar. > > > > > >Greg Bronevetsky > > >Post-Doctoral Researcher > > >1028 Building 451 > > >Lawrence Livermore National Lab > > >(925) 424-5756 > > >bronevetsky1(a)llnl.gov > > >http:// greg.bronevetsky.com > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >--------------------------------------------------------------------- > > >Intel GmbH > > >Dornacher Strasse 1 > > >85622 Feldkirchen/Muenchen Germany > > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > > >VAT Registration No.: DE129385895 > > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > > > >This e-mail and any attachments may contain confidential material for > > >the sole use of the intended recipient(s). Any review or distribution > > >by others is strictly prohibited. If you are not the intended > > >recipient, please contact the sender and delete all copies. > > > > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Supalov, Alexander 19 Feb '09

19 Feb '09

Dear Thomas, Thank you. I do not propose a certain checkpointer. I just assume that the MPI and the checkpointer involved will find each other somehow (this may be a configuration option, dynamic detection, whatever), and then they will arrange for the proper interaction. The proposed calls are just marking the point in the program where the MPI is prepared for checkpointing (MPI_PREPARE_FOR_CHECKPOINT) and is ready to resume after checkpoint (MPI_RESTORE_AFTER_CHECKPOINT). Depending on what the MPI and the checkpointer agree to do, the actual scope of the data saved will vary. What is proposed, in essence, is a couple of brackets that act almost like MPI_Finalize (MPI_PREPARE_FOR_CHECKPOINT) and MPI_Init (MPI_RESTORE_AFTER_CHECKPOINT), with the reservations wrt connection re-establishment, data types, etc. that you so appropriately pointed out. Note that the MPI_PREPARE_FOR_CHECKPOINT may not need to do any disconnects or remove or relocate any MPI objects. This depends on the checkpointer involved and the agreement between it and the MPI. Likewise, the MPI_RESTORE_AFTER_CHECKPOINT may not need to restore any objects if there was not restart in between. This needs to be communicated to the MPI. One way would be thru the callbacks that would be called from within the MPI_PREPARE_FOR_CHECKPOINT to notify the checkpointer that the MPI is ready, and another callback, or the same callback with another argument (e.g., 0 for preparation, 1 for restoration) that should tell the MPI whether restart actually took place. This can be done thru the callback return code or something. These callbacks can be registered with the MPI thru, say, MPI_REGISTER_CHECKPOINT_CALLBACK or something. Another way would be to extend the closing brackets into two (MPI_CONTINUE_AFTER_CHECKPOINT for simple continuation, MPI_RESTORE_AFTER_RESTART for actual restart), and a query function to ask the checkpointer at hand whether restart happened. In that case the aforementioned callback would only notify the checkpointer of the respective events. Note also that what the checkpointer actually does is up to the checkpointer. So, this may be a WRITE statement, if this is all the application really needs. A lot of flesh is yet to be added to these bones (scenarios, synchronization, callbacks, etc.), but I think that the concept is basically sound. Best regards. Alexander -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Thomas Herault Sent: Wednesday, February 18, 2009 7:53 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward Le 18 fï¿½vr. 09 ï¿½ 13:34, Supalov, Alexander a ï¿½crit : > Thanks. Let me address your objections one by one below. > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected] > ] On Behalf Of Greg Bronevetsky > Sent: Wednesday, February 18, 2009 6:43 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > First, this API works only for the synch-and-stop protocol. It does > not support any other protocol and this will lead into bad places in > the future because this protocol scales poorly with increasing > failure rates since it requires all processes to participate in every > single recovery. I can discuss the reasons for my assertion in more > detail if you'd like. > > AS> What other protocols are you talking about? I did not tell > anywhere whether these calls synch anything, or whether they are > collective, or whether the checkpointer is called on all nodes. This > remains to be defined. If you would please define other scenarios, > including those that scale well, I bet we'd probably be able to map > them to these calls that way or another. > > Second, this API is not compatible with application-level > checkpointing. An application-level checkpointer can deal with > internal MPI state in one of two ways. One option is to checkpoint it > at a high-level by using only the existing MPI calls, performing the > appropriate tracking of messages at calls to MPI_Send and MPI_Recv. > The simplest example of this is to ensure that there is no > communication going on at the time of the checkpoint, and then save > each process' state immediately after calling MPI_Barrier(). If this > is the option used, there is no need for any API extensions. > > AS> One option for the application level checkpointing is to save > the apps specific state (those two vectors) without saving the state > of the MPI at all. This will work if the MPI state is quiet (no > connections, nothing), and the application can be sure that its > state will be good, and when it restarts, the MPI hitting the > MPI_restore_after_checkpoint will know how to re-establish all > connections, etc. I think this is more than an MPI_Barrier, isn't it? > Hello Alexander, Greg, one point that was raised in the last forum meeting is that part of the application state is related to the MPI library internal state: for example, does the application expect that the communicators it created still exist after the restart? What about datatypes, etc...? Restarting MPI library is more than just re-establishing the connections. And that is Greg's point: - either we believe that the checkpointing system will save something for the MPI library (like the internal representation of the communicators, and which handles are connected to them), and then what the checkpointer will save for the MPI library must be specified somewhere - or the checkpointing system saves nothing for the MPI library. Here we have two new approaches: - either the user cannot expect anything except what is defined after MPI_Init to be still working. This approach advocates a simple modification of the standard: allowing an application to call MPI_Finalize(); ... MPI_Init(); multiple times during its execution. It could checkpoint between MPI_Finalize and MPI_Init - or design some way for the MPI library to save its internal state somehow. This could be done, but many system-level checkpointing systems do not provide functionalities to exclude the MPI library from the checkpoint, so the question arises on what can a MPI library do to cope with a partial initialization (restored by an unknown system- level checkpointer), and a state to restore, saved somewhere? > The second option is for the application to somehow same MPI internal > data structures, which is what the MPI_Prepare_for_checkpoint call is > for: to make sure that all MPI state is in locations accessible by > the application. Where is that? Well, it depends on the operating > system and the hardware. Does the application need to checkpoint and > restore MPI's open file handles or shared memory regions or is MPI > supposed to let those go during MPI_Prepare_for_checkpoint? The point > is, all these details cannot be defined in the MPI spec and if this > is the case, the call will not be sufficiently well-defined to be > used by an application-level checkpointer. However, it can be used by > the application to initiate a system-level checkpoint since in this > case all those details will not be the application's problem. > > AS> As for the place where all relevant MPI data is to be stored at > this point in time, we should not care - the MPI implementation and > the checkpointer involved will agree on this when the MPI will be > configured to use this particular checkpointer, either statically or > dynamically. The checkpointer will know that the data is in the > "right" place when it gats control after the > MPI_prepare_for_checkpoint call. > -> So, in your solution, an MPI library will come with a specific checkpointing tool? Do you enforce all implementations to come with a checkpointing tool? Checkpointing tools (system-level at least) are OS and hardware dependent. This is an issue for portability. And one could argue that it is not the role of the MPI library to save the application state. Not in the approach we decided to take for fault tolerance in MPI-3. > Finally, it does support system-initiated checkpointing. If the > system wants to start a checkpoint in the middle of an MPI_Send() > call, this API provides no help. If we care about this use-case we > need to allow checkpoints to be taken at all times or provide a way > to register a callback that tells the system when a checkpoint can > next be taken. > > AS> I suppose this is "does not support", right? If so, let's simply > add this callback registration call, say, > MPI_REGISTER_CHECKPOINT_CALLBACK, that will define the call to be > made when the MPI_Prepare_for_checkpoint is about to return control > to the application. This will I guess address your concern, won't it? > > The bottom line given the above issues is that the only benefit this > API does have is providing a common set of names for functions that > have poor semantics and limited functionality. This benefit does not > pass the bar. > > AS> See above. I think you raise valid points, but all of them can > be adequately addressed given the desire to provide the checkpoint- > restart capability > I agree with you. But I ask how far away from the initial goal we are willing to go, in order to provide checkpoint/restart capability? If the MPI library must embed (or be configured for) a checkpointing mechanism, we do not need to expose any API, or modify the existing standard: this has been done by many MPI implementations for multiple specific checkpointers. In this case, you do not need to expose anything to the application and do everything internally in the library. > If I'm allowed to digress here and refer back to the fault > notification, note that the checkpoint/restart capability is > something that does guarantee fault tolerance, also in the case when > fatal unrecoverable errors occur. You "simply" fix the machine and > restart from the earlier checkpoint. > > As such, this widely used and proven capability may be a safe first > bet for the FT WG to introduce as something people will like and > will use. Much more likely, in fact, than the notification stuff > that yet needs to be properly defined, and as currently envisioned, > with all that uncertain communicator restoration stuff, will require > substantial application rethinking and repgrogramming to be used. > > After all, this is only a matter of priorities. Going for a low > hanging fruit like checkpoint/restart may be unromantic, but it's > definitely very pragmatic and sound. And in any case, I call upon > the same set of criteria to be used for all things that we consider. > Usability in the sense that a feature will be used by commercial > apps is definitely one of them. Or so I think. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > http://greg.bronevetsky.com > Thomas > At 09:27 AM 2/18/2009, Supalov, Alexander wrote: >> I see. Let me explain what I think those two calls would do to >> facilitate checkpointing. >> >> The program piece would look as follows: >> >> CALL MPI_PREPARE_FOR_CHECKPOINT >> >> ! Whatever calls are necessary to do the actual checkpointing, >> including, in the case of resident monitoring checkpointer, none. >> >> CALL MPI_RESTORE_AFTER_CHECKPOINT >> >> The first call will bring MPI into a state that is acceptable to the >> checkpointer involved. This may include quiescence, etc. >> >> Then the checkpointer will do its job. Note that this may be >> application level checkpointer that saves two important arrays. Or >> this may be a system checkpointer that dumps the memory, etc. >> Finally, this can be a checkpointer that is notified when the >> MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in >> some way without the program calling it. >> >> The second call will restore the MPI state to what it was, >> functionally, after the checkpoint. "Functionally" here means that >> this may include running on a different set of nodes, etc. >> >> Is it this scenario that you think does not pass the bar? If so, in >> what way? >> >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org >> [mailto:[email protected]] On Behalf Of Greg >> Bronevetsky >> Sent: Wednesday, February 18, 2009 6:20 PM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >> Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working >> Group >> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >> >> My point is that each MPI implementation will have to negotiate with >> each checkpointer what this call will mean on each platform, meaning >> that it is upto the implementation-specific call to do all that >> magic. It doesn't save the application any real effort to call this >> routine by its implementation-specific name rather than a unified >> name given that the other two major problems are not solved: >> - Providing a uniform way for checkpointers and MPI implementations >> to interact (THE major for having this proposal in the first place) >> and >> - Support for more than the synch-and-stop checkpointing protocol, >> which does scales poorly with rising failure rates. >> >> My point here is that covering one of the three major issues is not >> good enough to modify the MPI specification and I don't think that >> we'll ever be able to do more than that. We're pretty much agreed in >> the working group that the first point above cannot be done and I >> don't think that the second can be done without providing callbacks >> to be called for each incoming/outgoing message. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> http:// greg.bronevetsky.com >> >> At 08:40 AM 2/18/2009, Supalov, Alexander wrote: >>> Thanks. How will the checkpointer-specific call talk to the MPI? >>> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org >>> [mailto:[email protected]] On Behalf Of Greg >>> Bronevetsky >>> Sent: Wednesday, February 18, 2009 5:29 PM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group >>> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >>> >>> >>>> Thanks. How will you let the MPI know the checkpoint is coming, to >>>> give it a fair chance to prepare to this and then recover after the >>>> checkpoint? This is akin to the MPI_Finalize/MPI_Init in some >>>> sense, >>>> midway thru the job, hence the analogy. >>> >>> Just use the checkpointer-specific call. The call is going to have >>> checkpointer-specific semantics, so why not give it a >>> checkpointer-specific name? I understand that there is some use to >>> allowing applications to use the same name across all checkpointers >>> but the bar should be higher than that for adding something to the >>> standard. Also, right now the whole approach inherently only >>> supports >>> one checkpointing protocol: synch-and-stop. If we can work out a >>> more >>> generic API that supports other protocols I think that it may have >>> enough value to be included in the spec. Right now it still hasn't >>> passed the bar. >>> >>> Greg Bronevetsky >>> Post-Doctoral Researcher >>> 1028 Building 451 >>> Lawrence Livermore National Lab >>> (925) 424-5756 >>> bronevetsky1(a)llnl.gov >>> http:// greg.bronevetsky.com >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> --------------------------------------------------------------------- >>> Intel GmbH >>> Dornacher Strasse 1 >>> 85622 Feldkirchen/Muenchen Germany >>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>> Registergericht: Muenchen HRB 47456 Ust.-IdNr. >>> VAT Registration No.: DE129385895 >>> Citibank Frankfurt (BLZ 502 109 00) 600119052 >>> >>> This e-mail and any attachments may contain confidential material >>> for >>> the sole use of the intended recipient(s). Any review or >>> distribution >>> by others is strictly prohibited. If you are not the intended >>> recipient, please contact the sender and delete all copies. >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> --------------------------------------------------------------------- >> Intel GmbH >> Dornacher Strasse 1 >> 85622 Feldkirchen/Muenchen Germany >> Sitz der Gesellschaft: Feldkirchen bei Muenchen >> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >> Registergericht: Muenchen HRB 47456 Ust.-IdNr. >> VAT Registration No.: DE129385895 >> Citibank Frankfurt (BLZ 502 109 00) 600119052 >> >> This e-mail and any attachments may contain confidential material for >> the sole use of the intended recipient(s). Any review or distribution >> by others is strictly prohibited. If you are not the intended >> recipient, please contact the sender and delete all copies. >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] MPI Fault Tolerance scenarios
by Erez Haba 18 Feb '09

18 Feb '09

I've posted this scenario on the FT wiki pages https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/scenarios_and_solutions From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Erez Haba Sent: Tuesday, February 17, 2009 6:53 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] MPI Fault Tolerance scenarios Hello all, In our last meeting we decided to build a set of FT scenarios/programs to help us understand the details of the interface need to support those scenarios. We also decided to start with very simple scenarios and add more complex ones as we understand the former better. I hope that starting with simple scenarios will help us build a solid foundation on which we can build the more complex solutions. When we build an FT solution we will focus on the scenario as described, without complicating the solution just because it would be needed later for a more complex one. The time will come later to modify the solution as we acquire more knowledge and built the foundations. Hence, any proposal or change that we make needs to fit exactly the scenario (and all those that we previously looked at) but no more. For example in the first scenario that we'll look at there is no need for saving communicator state or error callback; but they might be required later. Note that these scenarios focus on process FT rather than checkpoint/restart or network degradation. I assume we'll do the latter later. Scenario #1: Very Simple Master-Workers Description This is a very simple master-workers scenario. However simple, we were asked many times by customers to support FT in this scenario. In this case the MPI application running with n processes, where rank 0 is used as the master and n-1 ranks are used as workers. The master generates work (either by getting it directly from user input, or reading a file) and sends it for processing to a free worker rank. The master sends requests and receives replies using MPI point-to-point communication. The workers wait for the incoming message, upon arrival the worker computes the result and sends it back to the master. The master stores the result to a log file. Hardening: The goal is to harden the workers, the master itself is not FT, thus if it fails the entire application fails. In this case the workers are FT, and are replaced to keep computation power for this application. (a twist: if a worker cannot be recovered the master can work with a smaller set of clients up to a low watermark). Worker The worker waits on a blocking receive when a message arrives it process it. If a done message arrives the worker finalizes MPI and exit normally. Hardening: There is not special requirement for hardening here. If the worker encounters a communication problem with the master, it means that the master is down and it's okay to abort the entire job. Thus, it will use the default error handler (which aborts on errors). Note that we do not need to modify the client at all to make the application FT (except the master). Pseudo code for the hardened worker: int main() { MPI_Init() for(;;) { MPI_Recv(src=0, &query, MPI_COMM_WORLD); if(is_done_msg(query)) break; process_query(&query, &answer); MPI_Send(dst=0, &answer, MPI_COMM_WORLD); } MPI_Finalize() } Notice that for this FT code there is no requirement for the worker to rejoin the comm. As the only communicator used is MPI_COMM_WORLD. Master The master code reads queries from a stream and passes them on to the workers to process. The master goes through several phases. In the initialization phase it sends the first request to each one of the ranks; in the second one it shuts down any unnecessary ranks (if the job is too small); I the third phase it enters its progress engine where it handles replies (answers), process recovery and termination (on input end). Hardening: It is the responsibility of the master to restart any failing workers and make sure that the request (query) did not get lost if a worker fails. Hence, every time an error is detected the master will move the worker into repairing state and move its workload to other workers. The master runs with errors returned rather than aborted One thing to note about the following code: it is not optimized. I did not try to overlap computation with communication (which is possible) I tried to keep it as simple as possible for the purpose of discussion. Pseudo code for the hardened master; the code needed for repairing the failed ranks is highlighted in yellow. int main() { MPI_Init() >>> MPI_Comm_set_errhandler(MPI_COMM_WORLD, MPI_ERRORS_RETURN); MPI_Comm_size(MPI_COMM_WORLD, &n); MPI_Request r[n] = MPI_REQUEST_NULL; QueryMessage q[n]; AnswerMessage a[n]; int active_workers = 0; >>> bool repairing[n] = false; // // Phase 1: send initial requests // for(int i = 1; i < n; i++) { if(get_next_query(stream, &q[i]) == eof) break; active_workers++; MPI_Send(dest=i, &q[i], MPI_COMM_WORLD); rc = MPI_Irecv(src=i, buffer=&a[x], request=&r[x], MPI_COMM_WORLD) >>> if(rc != MPI_SUCCESS) >>> { >>> start_repair(i, repairing, q, a, r, stream); >>> } } // // Phase 2: finalize any unnecessary ranks // for(int i = active_workers + 1; i < n; i++) { MPI_Send(dest=i, &done_msg, MPI_COMM_WORLD); } // // The progress engine. Get answers; send new requests and handle // process repairs // while(active_workers != 0) { rc = MPI_Waitany(n, r, &i, MPI_STATUS_IGNORE); >>> if(!repairing[i]) >>> { >>> if(rc != MPI_SUCCESS) >>> { >>> start_repair(i, repairing, q, a, r, stream) >>> continue; >>> } process_answer(&a[i]); >>> } >>> else if(rc != MPI_SUCCESS) >>> { >>> active_workers--; >>> { if(get_next_input(stream, &q[i]) == eof) { active_workers--; MPI_Send(dest=i, &done_msg) { else { MPI_Send(dest=i, &q[i]) rc = MPI_Irecv(src=i, buffer=&a[i], request=&r[i], MPI_COMM_WORLD) >>> if(rc != MPI_SUCCESS) >>> { >>> start_repair(i, repairing, q, a, r, stream); >>> } } } MPI_Finalize() } >>> void start_repair(int i, int repairing[], Query q[], Answer q[], MPI_Request r[], Stream stream) >>> { >>> repairing[i] = true; >>> push_query_back(stream, &q[i]); >>> MPI_Comm_Irepair(MPI_COMM_WORLD, i, &r[i]); >>> } Logic description (without FT) The master code keeps track of the number of active workers through the active_workers variable. It is solely used for the purpose of shutdown. When the master is out of input, it shuts-down the workers by sending them 'done' message. It decrease the number of active workers and finalizes when this number reaches zero. The master's progress engine waits on a vector of requests (note that entry 0 is not used, as to simplify the code); one it gets an answer it processes it and sends the next query to that worker until it's out of input. Logic description (with FT) The master detects a faulty client either synchronously when it ties to initiate an async receive (no need to check the send, the assumption is that if send failed, so will the receive call), or async when the async receive completes with an error. Once an error detected (and identified as a faulty client, more about this later), the master starts an async repair of that client. If the repair succeeds, new work is sent to that client. If it does not, the number of active workers is decreased and the master has to live with less processing power. The code above assumes that if the returned code is an error, it should repair the worker; however as we discussed, there could very well be many different reasons for an error here, which not all are related to process failure; for that we might use something in lines of if(MPI_Error_event(rc) == MPI_EVENT_PROCESS_DOWN)... it would be the responsibility of the MPI implementation to encode or store the event related to the returned error code. (Note: in MPICH2 there is a mechanism that enables encoding extended error information in the error code, which then can be retrieved using MPI_Error_string) Conclusions I believe that the solution above describes what we have discussed in the last meeting. The required API's to support this FT are really minimal but already cover a good set of users. Please, send your comments. Thoughts? Thanks, .Erez P.S. I will post this on the FT wiki pages (with the feedbac). P.P.S. there is one more scenario that we discussed, and extension of the master-workers model. I will try to get it write us as-soon-as-posible.

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Greg Bronevetsky 18 Feb '09

18 Feb '09

>First, this API works only for the synch-and-stop protocol. It does >not support any other protocol and this will lead into bad places in >the future because this protocol scales poorly with increasing >failure rates since it requires all processes to participate in every >single recovery. I can discuss the reasons for my assertion in more >detail if you'd like. > >AS> What other protocols are you talking about? I did not tell >anywhere whether these calls synch anything, or whether they are >collective, or whether the checkpointer is called on all nodes. This >remains to be defined. If you would please define other scenarios, >including those that scale well, I bet we'd probably be able to map >them to these calls that way or another. Ok, lets look as message logging. Each process checkpoints independently but logs all non-deterministic events and the data of sent or received messages (details and overheads depend on protocol). If a process checkpoints, we can erase all the logged non-determinism or message data that relates to events before this checkpoint. If some process fails, it rolls back to its last checkpoint and all of the other processes resend to 1. the data of all messages they sent to it since that checkpoint and 2. the outcomes of all the nonderministic events that it experienced since that checkpoint. I know how to implement this either inside MPI or above MPI with the aid for the fault notification API. These things already exist, for example in MPICH-V. I can also imagine an API that lets the application ask MPI to run this protocol on its behalf. However, this API will not require quiescence or stable states, or anything like that. In particular, quiescence is completely incompatible with message logging. Realistically speaking, this procotol can only be provided to the application without any direct interaction with the application. This is fine but it also suggests that we don't need much of an API to support it. The same applies to other protocols that need to communicate before, after and/or during the checkpoint. As for your other points, Thomas' email covers roughly the same points that I would have made. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov http://greg.bronevetsky.com >Second, this API is not compatible with application-level >checkpointing. An application-level checkpointer can deal with >internal MPI state in one of two ways. One option is to checkpoint it >at a high-level by using only the existing MPI calls, performing the >appropriate tracking of messages at calls to MPI_Send and MPI_Recv. >The simplest example of this is to ensure that there is no >communication going on at the time of the checkpoint, and then save >each process' state immediately after calling MPI_Barrier(). If this >is the option used, there is no need for any API extensions. > >AS> One option for the application level checkpointing is to save >the apps specific state (those two vectors) without saving the state >of the MPI at all. This will work if the MPI state is quiet (no >connections, nothing), and the application can be sure that its >state will be good, and when it restarts, the MPI hitting the >MPI_restore_after_checkpoint will know how to re-establish all >connections, etc. I think this is more than an MPI_Barrier, isn't it? > >The second option is for the application to somehow same MPI internal >data structures, which is what the MPI_Prepare_for_checkpoint call is >for: to make sure that all MPI state is in locations accessible by >the application. Where is that? Well, it depends on the operating >system and the hardware. Does the application need to checkpoint and >restore MPI's open file handles or shared memory regions or is MPI >supposed to let those go during MPI_Prepare_for_checkpoint? The point >is, all these details cannot be defined in the MPI spec and if this >is the case, the call will not be sufficiently well-defined to be >used by an application-level checkpointer. However, it can be used by >the application to initiate a system-level checkpoint since in this >case all those details will not be the application's problem. > >AS> As for the place where all relevant MPI data is to be stored at >this point in time, we should not care - the MPI implementation and >the checkpointer involved will agree on this when the MPI will be >configured to use this particular checkpointer, either statically or >dynamically. The checkpointer will know that the data is in the >"right" place when it gats control after the MPI_prepare_for_checkpoint call. > >Finally, it does support system-initiated checkpointing. If the >system wants to start a checkpoint in the middle of an MPI_Send() >call, this API provides no help. If we care about this use-case we >need to allow checkpoints to be taken at all times or provide a way >to register a callback that tells the system when a checkpoint can >next be taken. > >AS> I suppose this is "does not support", right? If so, let's simply >add this callback registration call, say, >MPI_REGISTER_CHECKPOINT_CALLBACK, that will define the call to be >made when the MPI_Prepare_for_checkpoint is about to return control >to the application. This will I guess address your concern, won't it? > >The bottom line given the above issues is that the only benefit this >API does have is providing a common set of names for functions that >have poor semantics and limited functionality. This benefit does not >pass the bar. > >AS> See above. I think you raise valid points, but all of them can >be adequately addressed given the desire to provide the >checkpoint-restart capability > >If I'm allowed to digress here and refer back to the fault >notification, note that the checkpoint/restart capability is >something that does guarantee fault tolerance, also in the case when >fatal unrecoverable errors occur. You "simply" fix the machine and >restart from the earlier checkpoint. > >As such, this widely used and proven capability may be a safe first >bet for the FT WG to introduce as something people will like and >will use. Much more likely, in fact, than the notification stuff >that yet needs to be properly defined, and as currently envisioned, >with all that uncertain communicator restoration stuff, will require >substantial application rethinking and repgrogramming to be used. > >After all, this is only a matter of priorities. Going for a low >hanging fruit like checkpoint/restart may be unromantic, but it's >definitely very pragmatic and sound. And in any case, I call upon >the same set of criteria to be used for all things that we consider. >Usability in the sense that a feature will be used by commercial >apps is definitely one of them. Or so I think. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov >http:// greg.bronevetsky.com > >At 09:27 AM 2/18/2009, Supalov, Alexander wrote: > >I see. Let me explain what I think those two calls would do to > >facilitate checkpointing. > > > >The program piece would look as follows: > > > >CALL MPI_PREPARE_FOR_CHECKPOINT > > > >! Whatever calls are necessary to do the actual checkpointing, > >including, in the case of resident monitoring checkpointer, none. > > > >CALL MPI_RESTORE_AFTER_CHECKPOINT > > > >The first call will bring MPI into a state that is acceptable to the > >checkpointer involved. This may include quiescence, etc. > > > >Then the checkpointer will do its job. Note that this may be > >application level checkpointer that saves two important arrays. Or > >this may be a system checkpointer that dumps the memory, etc. > >Finally, this can be a checkpointer that is notified when the > >MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in > >some way without the program calling it. > > > >The second call will restore the MPI state to what it was, > >functionally, after the checkpoint. "Functionally" here means that > >this may include running on a different set of nodes, etc. > > > >Is it this scenario that you think does not pass the bar? If so, > in what way? > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > >Sent: Wednesday, February 18, 2009 6:20 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > >My point is that each MPI implementation will have to negotiate with > >each checkpointer what this call will mean on each platform, meaning > >that it is upto the implementation-specific call to do all that > >magic. It doesn't save the application any real effort to call this > >routine by its implementation-specific name rather than a unified > >name given that the other two major problems are not solved: > >- Providing a uniform way for checkpointers and MPI implementations > >to interact (THE major for having this proposal in the first place) and > >- Support for more than the synch-and-stop checkpointing protocol, > >which does scales poorly with rising failure rates. > > > >My point here is that covering one of the three major issues is not > >good enough to modify the MPI specification and I don't think that > >we'll ever be able to do more than that. We're pretty much agreed in > >the working group that the first point above cannot be done and I > >don't think that the second can be done without providing callbacks > >to be called for each incoming/outgoing message. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > >http:// greg.bronevetsky.com > > > >At 08:40 AM 2/18/2009, Supalov, Alexander wrote: > > >Thanks. How will the checkpointer-specific call talk to the MPI? > > > > > >-----Original Message----- > > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > > >Sent: Wednesday, February 18, 2009 5:29 PM > > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > > > > > > > >Thanks. How will you let the MPI know the checkpoint is coming, to > > > >give it a fair chance to prepare to this and then recover after the > > > >checkpoint? This is akin to the MPI_Finalize/MPI_Init in some sense, > > > >midway thru the job, hence the analogy. > > > > > >Just use the checkpointer-specific call. The call is going to have > > >checkpointer-specific semantics, so why not give it a > > >checkpointer-specific name? I understand that there is some use to > > >allowing applications to use the same name across all checkpointers > > >but the bar should be higher than that for adding something to the > > >standard. Also, right now the whole approach inherently only supports > > >one checkpointing protocol: synch-and-stop. If we can work out a more > > >generic API that supports other protocols I think that it may have > > >enough value to be included in the spec. Right now it still hasn't > > >passed the bar. > > > > > >Greg Bronevetsky > > >Post-Doctoral Researcher > > >1028 Building 451 > > >Lawrence Livermore National Lab > > >(925) 424-5756 > > >bronevetsky1(a)llnl.gov > > >http:// greg.bronevetsky.com > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >--------------------------------------------------------------------- > > >Intel GmbH > > >Dornacher Strasse 1 > > >85622 Feldkirchen/Muenchen Germany > > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > > >VAT Registration No.: DE129385895 > > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > > > >This e-mail and any attachments may contain confidential material for > > >the sole use of the intended recipient(s). Any review or distribution > > >by others is strictly prohibited. If you are not the intended > > >recipient, please contact the sender and delete all copies. > > > > > > > > >_______________________________________________ > > >mpi3-ft mailing list > > >mpi3-ft(a)lists.mpi-forum.org > > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Thomas Herault 18 Feb '09

18 Feb '09

Le 18 fï¿½vr. 09 ï¿½ 13:34, Supalov, Alexander a ï¿½crit : > Thanks. Let me address your objections one by one below. > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected] > ] On Behalf Of Greg Bronevetsky > Sent: Wednesday, February 18, 2009 6:43 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group > Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > First, this API works only for the synch-and-stop protocol. It does > not support any other protocol and this will lead into bad places in > the future because this protocol scales poorly with increasing > failure rates since it requires all processes to participate in every > single recovery. I can discuss the reasons for my assertion in more > detail if you'd like. > > AS> What other protocols are you talking about? I did not tell > anywhere whether these calls synch anything, or whether they are > collective, or whether the checkpointer is called on all nodes. This > remains to be defined. If you would please define other scenarios, > including those that scale well, I bet we'd probably be able to map > them to these calls that way or another. > > Second, this API is not compatible with application-level > checkpointing. An application-level checkpointer can deal with > internal MPI state in one of two ways. One option is to checkpoint it > at a high-level by using only the existing MPI calls, performing the > appropriate tracking of messages at calls to MPI_Send and MPI_Recv. > The simplest example of this is to ensure that there is no > communication going on at the time of the checkpoint, and then save > each process' state immediately after calling MPI_Barrier(). If this > is the option used, there is no need for any API extensions. > > AS> One option for the application level checkpointing is to save > the apps specific state (those two vectors) without saving the state > of the MPI at all. This will work if the MPI state is quiet (no > connections, nothing), and the application can be sure that its > state will be good, and when it restarts, the MPI hitting the > MPI_restore_after_checkpoint will know how to re-establish all > connections, etc. I think this is more than an MPI_Barrier, isn't it? > Hello Alexander, Greg, one point that was raised in the last forum meeting is that part of the application state is related to the MPI library internal state: for example, does the application expect that the communicators it created still exist after the restart? What about datatypes, etc...? Restarting MPI library is more than just re-establishing the connections. And that is Greg's point: - either we believe that the checkpointing system will save something for the MPI library (like the internal representation of the communicators, and which handles are connected to them), and then what the checkpointer will save for the MPI library must be specified somewhere - or the checkpointing system saves nothing for the MPI library. Here we have two new approaches: - either the user cannot expect anything except what is defined after MPI_Init to be still working. This approach advocates a simple modification of the standard: allowing an application to call MPI_Finalize(); ... MPI_Init(); multiple times during its execution. It could checkpoint between MPI_Finalize and MPI_Init - or design some way for the MPI library to save its internal state somehow. This could be done, but many system-level checkpointing systems do not provide functionalities to exclude the MPI library from the checkpoint, so the question arises on what can a MPI library do to cope with a partial initialization (restored by an unknown system- level checkpointer), and a state to restore, saved somewhere? > The second option is for the application to somehow same MPI internal > data structures, which is what the MPI_Prepare_for_checkpoint call is > for: to make sure that all MPI state is in locations accessible by > the application. Where is that? Well, it depends on the operating > system and the hardware. Does the application need to checkpoint and > restore MPI's open file handles or shared memory regions or is MPI > supposed to let those go during MPI_Prepare_for_checkpoint? The point > is, all these details cannot be defined in the MPI spec and if this > is the case, the call will not be sufficiently well-defined to be > used by an application-level checkpointer. However, it can be used by > the application to initiate a system-level checkpoint since in this > case all those details will not be the application's problem. > > AS> As for the place where all relevant MPI data is to be stored at > this point in time, we should not care - the MPI implementation and > the checkpointer involved will agree on this when the MPI will be > configured to use this particular checkpointer, either statically or > dynamically. The checkpointer will know that the data is in the > "right" place when it gats control after the > MPI_prepare_for_checkpoint call. > -> So, in your solution, an MPI library will come with a specific checkpointing tool? Do you enforce all implementations to come with a checkpointing tool? Checkpointing tools (system-level at least) are OS and hardware dependent. This is an issue for portability. And one could argue that it is not the role of the MPI library to save the application state. Not in the approach we decided to take for fault tolerance in MPI-3. > Finally, it does support system-initiated checkpointing. If the > system wants to start a checkpoint in the middle of an MPI_Send() > call, this API provides no help. If we care about this use-case we > need to allow checkpoints to be taken at all times or provide a way > to register a callback that tells the system when a checkpoint can > next be taken. > > AS> I suppose this is "does not support", right? If so, let's simply > add this callback registration call, say, > MPI_REGISTER_CHECKPOINT_CALLBACK, that will define the call to be > made when the MPI_Prepare_for_checkpoint is about to return control > to the application. This will I guess address your concern, won't it? > > The bottom line given the above issues is that the only benefit this > API does have is providing a common set of names for functions that > have poor semantics and limited functionality. This benefit does not > pass the bar. > > AS> See above. I think you raise valid points, but all of them can > be adequately addressed given the desire to provide the checkpoint- > restart capability > I agree with you. But I ask how far away from the initial goal we are willing to go, in order to provide checkpoint/restart capability? If the MPI library must embed (or be configured for) a checkpointing mechanism, we do not need to expose any API, or modify the existing standard: this has been done by many MPI implementations for multiple specific checkpointers. In this case, you do not need to expose anything to the application and do everything internally in the library. > If I'm allowed to digress here and refer back to the fault > notification, note that the checkpoint/restart capability is > something that does guarantee fault tolerance, also in the case when > fatal unrecoverable errors occur. You "simply" fix the machine and > restart from the earlier checkpoint. > > As such, this widely used and proven capability may be a safe first > bet for the FT WG to introduce as something people will like and > will use. Much more likely, in fact, than the notification stuff > that yet needs to be properly defined, and as currently envisioned, > with all that uncertain communicator restoration stuff, will require > substantial application rethinking and repgrogramming to be used. > > After all, this is only a matter of priorities. Going for a low > hanging fruit like checkpoint/restart may be unromantic, but it's > definitely very pragmatic and sound. And in any case, I call upon > the same set of criteria to be used for all things that we consider. > Usability in the sense that a feature will be used by commercial > apps is definitely one of them. Or so I think. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > http://greg.bronevetsky.com > Thomas > At 09:27 AM 2/18/2009, Supalov, Alexander wrote: >> I see. Let me explain what I think those two calls would do to >> facilitate checkpointing. >> >> The program piece would look as follows: >> >> CALL MPI_PREPARE_FOR_CHECKPOINT >> >> ! Whatever calls are necessary to do the actual checkpointing, >> including, in the case of resident monitoring checkpointer, none. >> >> CALL MPI_RESTORE_AFTER_CHECKPOINT >> >> The first call will bring MPI into a state that is acceptable to the >> checkpointer involved. This may include quiescence, etc. >> >> Then the checkpointer will do its job. Note that this may be >> application level checkpointer that saves two important arrays. Or >> this may be a system checkpointer that dumps the memory, etc. >> Finally, this can be a checkpointer that is notified when the >> MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in >> some way without the program calling it. >> >> The second call will restore the MPI state to what it was, >> functionally, after the checkpoint. "Functionally" here means that >> this may include running on a different set of nodes, etc. >> >> Is it this scenario that you think does not pass the bar? If so, in >> what way? >> >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org >> [mailto:[email protected]] On Behalf Of Greg >> Bronevetsky >> Sent: Wednesday, February 18, 2009 6:20 PM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >> Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working >> Group >> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >> >> My point is that each MPI implementation will have to negotiate with >> each checkpointer what this call will mean on each platform, meaning >> that it is upto the implementation-specific call to do all that >> magic. It doesn't save the application any real effort to call this >> routine by its implementation-specific name rather than a unified >> name given that the other two major problems are not solved: >> - Providing a uniform way for checkpointers and MPI implementations >> to interact (THE major for having this proposal in the first place) >> and >> - Support for more than the synch-and-stop checkpointing protocol, >> which does scales poorly with rising failure rates. >> >> My point here is that covering one of the three major issues is not >> good enough to modify the MPI specification and I don't think that >> we'll ever be able to do more than that. We're pretty much agreed in >> the working group that the first point above cannot be done and I >> don't think that the second can be done without providing callbacks >> to be called for each incoming/outgoing message. >> >> Greg Bronevetsky >> Post-Doctoral Researcher >> 1028 Building 451 >> Lawrence Livermore National Lab >> (925) 424-5756 >> bronevetsky1(a)llnl.gov >> http:// greg.bronevetsky.com >> >> At 08:40 AM 2/18/2009, Supalov, Alexander wrote: >>> Thanks. How will the checkpointer-specific call talk to the MPI? >>> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org >>> [mailto:[email protected]] On Behalf Of Greg >>> Bronevetsky >>> Sent: Wednesday, February 18, 2009 5:29 PM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working >>> Group >>> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward >>> >>> >>>> Thanks. How will you let the MPI know the checkpoint is coming, to >>>> give it a fair chance to prepare to this and then recover after the >>>> checkpoint? This is akin to the MPI_Finalize/MPI_Init in some >>>> sense, >>>> midway thru the job, hence the analogy. >>> >>> Just use the checkpointer-specific call. The call is going to have >>> checkpointer-specific semantics, so why not give it a >>> checkpointer-specific name? I understand that there is some use to >>> allowing applications to use the same name across all checkpointers >>> but the bar should be higher than that for adding something to the >>> standard. Also, right now the whole approach inherently only >>> supports >>> one checkpointing protocol: synch-and-stop. If we can work out a >>> more >>> generic API that supports other protocols I think that it may have >>> enough value to be included in the spec. Right now it still hasn't >>> passed the bar. >>> >>> Greg Bronevetsky >>> Post-Doctoral Researcher >>> 1028 Building 451 >>> Lawrence Livermore National Lab >>> (925) 424-5756 >>> bronevetsky1(a)llnl.gov >>> http:// greg.bronevetsky.com >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> --------------------------------------------------------------------- >>> Intel GmbH >>> Dornacher Strasse 1 >>> 85622 Feldkirchen/Muenchen Germany >>> Sitz der Gesellschaft: Feldkirchen bei Muenchen >>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >>> Registergericht: Muenchen HRB 47456 Ust.-IdNr. >>> VAT Registration No.: DE129385895 >>> Citibank Frankfurt (BLZ 502 109 00) 600119052 >>> >>> This e-mail and any attachments may contain confidential material >>> for >>> the sole use of the intended recipient(s). Any review or >>> distribution >>> by others is strictly prohibited. If you are not the intended >>> recipient, please contact the sender and delete all copies. >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> --------------------------------------------------------------------- >> Intel GmbH >> Dornacher Strasse 1 >> 85622 Feldkirchen/Muenchen Germany >> Sitz der Gesellschaft: Feldkirchen bei Muenchen >> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >> Registergericht: Muenchen HRB 47456 Ust.-IdNr. >> VAT Registration No.: DE129385895 >> Citibank Frankfurt (BLZ 502 109 00) 600119052 >> >> This e-mail and any attachments may contain confidential material for >> the sole use of the intended recipient(s). Any review or distribution >> by others is strictly prohibited. If you are not the intended >> recipient, please contact the sender and delete all copies. >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Supalov, Alexander 18 Feb '09

18 Feb '09

Thanks. Let me address your objections one by one below. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Wednesday, February 18, 2009 6:43 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward First, this API works only for the synch-and-stop protocol. It does not support any other protocol and this will lead into bad places in the future because this protocol scales poorly with increasing failure rates since it requires all processes to participate in every single recovery. I can discuss the reasons for my assertion in more detail if you'd like. AS> What other protocols are you talking about? I did not tell anywhere whether these calls synch anything, or whether they are collective, or whether the checkpointer is called on all nodes. This remains to be defined. If you would please define other scenarios, including those that scale well, I bet we'd probably be able to map them to these calls that way or another. Second, this API is not compatible with application-level checkpointing. An application-level checkpointer can deal with internal MPI state in one of two ways. One option is to checkpoint it at a high-level by using only the existing MPI calls, performing the appropriate tracking of messages at calls to MPI_Send and MPI_Recv. The simplest example of this is to ensure that there is no communication going on at the time of the checkpoint, and then save each process' state immediately after calling MPI_Barrier(). If this is the option used, there is no need for any API extensions. AS> One option for the application level checkpointing is to save the apps specific state (those two vectors) without saving the state of the MPI at all. This will work if the MPI state is quiet (no connections, nothing), and the application can be sure that its state will be good, and when it restarts, the MPI hitting the MPI_restore_after_checkpoint will know how to re-establish all connections, etc. I think this is more than an MPI_Barrier, isn't it? The second option is for the application to somehow same MPI internal data structures, which is what the MPI_Prepare_for_checkpoint call is for: to make sure that all MPI state is in locations accessible by the application. Where is that? Well, it depends on the operating system and the hardware. Does the application need to checkpoint and restore MPI's open file handles or shared memory regions or is MPI supposed to let those go during MPI_Prepare_for_checkpoint? The point is, all these details cannot be defined in the MPI spec and if this is the case, the call will not be sufficiently well-defined to be used by an application-level checkpointer. However, it can be used by the application to initiate a system-level checkpoint since in this case all those details will not be the application's problem. AS> As for the place where all relevant MPI data is to be stored at this point in time, we should not care - the MPI implementation and the checkpointer involved will agree on this when the MPI will be configured to use this particular checkpointer, either statically or dynamically. The checkpointer will know that the data is in the "right" place when it gats control after the MPI_prepare_for_checkpoint call. Finally, it does support system-initiated checkpointing. If the system wants to start a checkpoint in the middle of an MPI_Send() call, this API provides no help. If we care about this use-case we need to allow checkpoints to be taken at all times or provide a way to register a callback that tells the system when a checkpoint can next be taken. AS> I suppose this is "does not support", right? If so, let's simply add this callback registration call, say, MPI_REGISTER_CHECKPOINT_CALLBACK, that will define the call to be made when the MPI_Prepare_for_checkpoint is about to return control to the application. This will I guess address your concern, won't it? The bottom line given the above issues is that the only benefit this API does have is providing a common set of names for functions that have poor semantics and limited functionality. This benefit does not pass the bar. AS> See above. I think you raise valid points, but all of them can be adequately addressed given the desire to provide the checkpoint-restart capability If I'm allowed to digress here and refer back to the fault notification, note that the checkpoint/restart capability is something that does guarantee fault tolerance, also in the case when fatal unrecoverable errors occur. You "simply" fix the machine and restart from the earlier checkpoint. As such, this widely used and proven capability may be a safe first bet for the FT WG to introduce as something people will like and will use. Much more likely, in fact, than the notification stuff that yet needs to be properly defined, and as currently envisioned, with all that uncertain communicator restoration stuff, will require substantial application rethinking and repgrogramming to be used. After all, this is only a matter of priorities. Going for a low hanging fruit like checkpoint/restart may be unromantic, but it's definitely very pragmatic and sound. And in any case, I call upon the same set of criteria to be used for all things that we consider. Usability in the sense that a feature will be used by commercial apps is definitely one of them. Or so I think. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov http://greg.bronevetsky.com At 09:27 AM 2/18/2009, Supalov, Alexander wrote: >I see. Let me explain what I think those two calls would do to >facilitate checkpointing. > >The program piece would look as follows: > >CALL MPI_PREPARE_FOR_CHECKPOINT > >! Whatever calls are necessary to do the actual checkpointing, >including, in the case of resident monitoring checkpointer, none. > >CALL MPI_RESTORE_AFTER_CHECKPOINT > >The first call will bring MPI into a state that is acceptable to the >checkpointer involved. This may include quiescence, etc. > >Then the checkpointer will do its job. Note that this may be >application level checkpointer that saves two important arrays. Or >this may be a system checkpointer that dumps the memory, etc. >Finally, this can be a checkpointer that is notified when the >MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in >some way without the program calling it. > >The second call will restore the MPI state to what it was, >functionally, after the checkpoint. "Functionally" here means that >this may include running on a different set of nodes, etc. > >Is it this scenario that you think does not pass the bar? If so, in what way? > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg Bronevetsky >Sent: Wednesday, February 18, 2009 6:20 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > >My point is that each MPI implementation will have to negotiate with >each checkpointer what this call will mean on each platform, meaning >that it is upto the implementation-specific call to do all that >magic. It doesn't save the application any real effort to call this >routine by its implementation-specific name rather than a unified >name given that the other two major problems are not solved: >- Providing a uniform way for checkpointers and MPI implementations >to interact (THE major for having this proposal in the first place) and >- Support for more than the synch-and-stop checkpointing protocol, >which does scales poorly with rising failure rates. > >My point here is that covering one of the three major issues is not >good enough to modify the MPI specification and I don't think that >we'll ever be able to do more than that. We're pretty much agreed in >the working group that the first point above cannot be done and I >don't think that the second can be done without providing callbacks >to be called for each incoming/outgoing message. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov >http:// greg.bronevetsky.com > >At 08:40 AM 2/18/2009, Supalov, Alexander wrote: > >Thanks. How will the checkpointer-specific call talk to the MPI? > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > >Sent: Wednesday, February 18, 2009 5:29 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > > > > >Thanks. How will you let the MPI know the checkpoint is coming, to > > >give it a fair chance to prepare to this and then recover after the > > >checkpoint? This is akin to the MPI_Finalize/MPI_Init in some sense, > > >midway thru the job, hence the analogy. > > > >Just use the checkpointer-specific call. The call is going to have > >checkpointer-specific semantics, so why not give it a > >checkpointer-specific name? I understand that there is some use to > >allowing applications to use the same name across all checkpointers > >but the bar should be higher than that for adding something to the > >standard. Also, right now the whole approach inherently only supports > >one checkpointing protocol: synch-and-stop. If we can work out a more > >generic API that supports other protocols I think that it may have > >enough value to be included in the spec. Right now it still hasn't > >passed the bar. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > >http:// greg.bronevetsky.com > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Greg Bronevetsky 18 Feb '09

18 Feb '09

First, this API works only for the synch-and-stop protocol. It does not support any other protocol and this will lead into bad places in the future because this protocol scales poorly with increasing failure rates since it requires all processes to participate in every single recovery. I can discuss the reasons for my assertion in more detail if you'd like. Second, this API is not compatible with application-level checkpointing. An application-level checkpointer can deal with internal MPI state in one of two ways. One option is to checkpoint it at a high-level by using only the existing MPI calls, performing the appropriate tracking of messages at calls to MPI_Send and MPI_Recv. The simplest example of this is to ensure that there is no communication going on at the time of the checkpoint, and then save each process' state immediately after calling MPI_Barrier(). If this is the option used, there is no need for any API extensions. The second option is for the application to somehow same MPI internal data structures, which is what the MPI_Prepare_for_checkpoint call is for: to make sure that all MPI state is in locations accessible by the application. Where is that? Well, it depends on the operating system and the hardware. Does the application need to checkpoint and restore MPI's open file handles or shared memory regions or is MPI supposed to let those go during MPI_Prepare_for_checkpoint? The point is, all these details cannot be defined in the MPI spec and if this is the case, the call will not be sufficiently well-defined to be used by an application-level checkpointer. However, it can be used by the application to initiate a system-level checkpoint since in this case all those details will not be the application's problem. Finally, it does support system-initiated checkpointing. If the system wants to start a checkpoint in the middle of an MPI_Send() call, this API provides no help. If we care about this use-case we need to allow checkpoints to be taken at all times or provide a way to register a callback that tells the system when a checkpoint can next be taken. The bottom line given the above issues is that the only benefit this API does have is providing a common set of names for functions that have poor semantics and limited functionality. This benefit does not pass the bar. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov http://greg.bronevetsky.com At 09:27 AM 2/18/2009, Supalov, Alexander wrote: >I see. Let me explain what I think those two calls would do to >facilitate checkpointing. > >The program piece would look as follows: > >CALL MPI_PREPARE_FOR_CHECKPOINT > >! Whatever calls are necessary to do the actual checkpointing, >including, in the case of resident monitoring checkpointer, none. > >CALL MPI_RESTORE_AFTER_CHECKPOINT > >The first call will bring MPI into a state that is acceptable to the >checkpointer involved. This may include quiescence, etc. > >Then the checkpointer will do its job. Note that this may be >application level checkpointer that saves two important arrays. Or >this may be a system checkpointer that dumps the memory, etc. >Finally, this can be a checkpointer that is notified when the >MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in >some way without the program calling it. > >The second call will restore the MPI state to what it was, >functionally, after the checkpoint. "Functionally" here means that >this may include running on a different set of nodes, etc. > >Is it this scenario that you think does not pass the bar? If so, in what way? > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg Bronevetsky >Sent: Wednesday, February 18, 2009 6:20 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > >My point is that each MPI implementation will have to negotiate with >each checkpointer what this call will mean on each platform, meaning >that it is upto the implementation-specific call to do all that >magic. It doesn't save the application any real effort to call this >routine by its implementation-specific name rather than a unified >name given that the other two major problems are not solved: >- Providing a uniform way for checkpointers and MPI implementations >to interact (THE major for having this proposal in the first place) and >- Support for more than the synch-and-stop checkpointing protocol, >which does scales poorly with rising failure rates. > >My point here is that covering one of the three major issues is not >good enough to modify the MPI specification and I don't think that >we'll ever be able to do more than that. We're pretty much agreed in >the working group that the first point above cannot be done and I >don't think that the second can be done without providing callbacks >to be called for each incoming/outgoing message. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov >http:// greg.bronevetsky.com > >At 08:40 AM 2/18/2009, Supalov, Alexander wrote: > >Thanks. How will the checkpointer-specific call talk to the MPI? > > > >-----Original Message----- > >From: mpi3-ft-bounces(a)lists.mpi-forum.org > >[mailto:[email protected]] On Behalf Of Greg Bronevetsky > >Sent: Wednesday, February 18, 2009 5:29 PM > >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working > >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > > > > >Thanks. How will you let the MPI know the checkpoint is coming, to > > >give it a fair chance to prepare to this and then recover after the > > >checkpoint? This is akin to the MPI_Finalize/MPI_Init in some sense, > > >midway thru the job, hence the analogy. > > > >Just use the checkpointer-specific call. The call is going to have > >checkpointer-specific semantics, so why not give it a > >checkpointer-specific name? I understand that there is some use to > >allowing applications to use the same name across all checkpointers > >but the bar should be higher than that for adding something to the > >standard. Also, right now the whole approach inherently only supports > >one checkpointing protocol: synch-and-stop. If we can work out a more > >generic API that supports other protocols I think that it may have > >enough value to be included in the spec. Right now it still hasn't > >passed the bar. > > > >Greg Bronevetsky > >Post-Doctoral Researcher > >1028 Building 451 > >Lawrence Livermore National Lab > >(925) 424-5756 > >bronevetsky1(a)llnl.gov > >http:// greg.bronevetsky.com > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >--------------------------------------------------------------------- > >Intel GmbH > >Dornacher Strasse 1 > >85622 Feldkirchen/Muenchen Germany > >Sitz der Gesellschaft: Feldkirchen bei Muenchen > >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > >Registergericht: Muenchen HRB 47456 Ust.-IdNr. > >VAT Registration No.: DE129385895 > >Citibank Frankfurt (BLZ 502 109 00) 600119052 > > > >This e-mail and any attachments may contain confidential material for > >the sole use of the intended recipient(s). Any review or distribution > >by others is strictly prohibited. If you are not the intended > >recipient, please contact the sender and delete all copies. > > > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Communicator Virtualization as a step forward
by Supalov, Alexander 18 Feb '09

18 Feb '09

I see. Let me explain what I think those two calls would do to facilitate checkpointing. The program piece would look as follows: CALL MPI_PREPARE_FOR_CHECKPOINT ! Whatever calls are necessary to do the actual checkpointing, including, in the case of resident monitoring checkpointer, none. CALL MPI_RESTORE_AFTER_CHECKPOINT The first call will bring MPI into a state that is acceptable to the checkpointer involved. This may include quiescence, etc. Then the checkpointer will do its job. Note that this may be application level checkpointer that saves two important arrays. Or this may be a system checkpointer that dumps the memory, etc. Finally, this can be a checkpointer that is notified when the MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in some way without the program calling it. The second call will restore the MPI state to what it was, functionally, after the checkpoint. "Functionally" here means that this may include running on a different set of nodes, etc. Is it this scenario that you think does not pass the bar? If so, in what way? -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Wednesday, February 18, 2009 6:20 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward My point is that each MPI implementation will have to negotiate with each checkpointer what this call will mean on each platform, meaning that it is upto the implementation-specific call to do all that magic. It doesn't save the application any real effort to call this routine by its implementation-specific name rather than a unified name given that the other two major problems are not solved: - Providing a uniform way for checkpointers and MPI implementations to interact (THE major for having this proposal in the first place) and - Support for more than the synch-and-stop checkpointing protocol, which does scales poorly with rising failure rates. My point here is that covering one of the three major issues is not good enough to modify the MPI specification and I don't think that we'll ever be able to do more than that. We're pretty much agreed in the working group that the first point above cannot be done and I don't think that the second can be done without providing callbacks to be called for each incoming/outgoing message. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov http://greg.bronevetsky.com At 08:40 AM 2/18/2009, Supalov, Alexander wrote: >Thanks. How will the checkpointer-specific call talk to the MPI? > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg Bronevetsky >Sent: Wednesday, February 18, 2009 5:29 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working >Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward > > > >Thanks. How will you let the MPI know the checkpoint is coming, to > >give it a fair chance to prepare to this and then recover after the > >checkpoint? This is akin to the MPI_Finalize/MPI_Init in some sense, > >midway thru the job, hence the analogy. > >Just use the checkpointer-specific call. The call is going to have >checkpointer-specific semantics, so why not give it a >checkpointer-specific name? I understand that there is some use to >allowing applications to use the same name across all checkpointers >but the bar should be higher than that for adding something to the >standard. Also, right now the whole approach inherently only supports >one checkpointing protocol: synch-and-stop. If we can work out a more >generic API that supports other protocols I think that it may have >enough value to be included in the spec. Right now it still hasn't >passed the bar. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov >http:// greg.bronevetsky.com > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0