October 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 24 Oct '08

24 Oct '08

PS. Regarding MPI state freeze: if the CR system involved can consistenly capture and restore global network and wire state in addition to the memory state, the proposed calls can be nearly no-ops. -----Original Message----- From: Supalov, Alexander Sent: Friday, October 24, 2008 11:04 PM To: 'MPI 3.0 Fault Tolerance and Dynamic Process Control working Group' Subject: RE: [Mpi3-ft] system-level C/R requirements Thanks. I can't speak for the whole Forum, but my impression is that if the choice will be between solving the problem of MPI and CR on one hand, and not solving it on the other hand, a reasonable proposal will go a long way toward convincing the majority, or at least moving the discussion to a still better proposal. As for the number of calls, this is question of ROI. We're going to add 200 or so fancy calls by the latest guess, while here we have just 2 that offer basic functionality of undeniable value. This should be acceptable. Finally, I don't know a more implementation specific call than MPI_Init. The proposed calls live close nearby. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Friday, October 24, 2008 10:51 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements >Probably we should not try to define this, and encourage >checkpoint/restart people work with particular MPI implementations to >make sure things work? After all, it's not about an abstract MPI and >abstract CR system - there will probably always be actual pairs (or >other relationships) of them that can work together. > >Compare this to the situation with threads. MPI acknowledges their >existence and provide a couple of calls to request a particular level of >support, that's all. This was good enough for starters, and may change >in the future. The relation between MPI and CR may go this way, too: >acknowledging first, integrating next. If all that you want to add to the spec is a couple of calls to say that MPI state should be frozen in some sense, that's fine with me since we're defining what these calls do. However, I'm not sure that the wider forum will accept them because there is a general dislike for adding more calls and the primary value of these calls is to motivate people to provide implementation-specific definitions for them, which I'm guessing is lower than the standard bar for acceptance. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Greg Bronevetsky 24 Oct '08

24 Oct '08

I think that the problem for the forum will be the unclear semantics of the new calls. MPI_Init is not a good example because it has clear semantics for all users of MPI but not system-level services. The difference with the quiscence calls is that we're trying to provide a way to by-pass to regular MPI semantics and plug into the middle of MPI without precisely defining how the by-pass works. Precise semantics didn't matter for MPI_Init exactly because there has never been a way to look into the MPI implementation until now. The solution to this is to provide very loose semantics to the new calls but this just means that there will actually be no standard way to use the new calls, which is why I'm afraid the forum will not like it. I can think of only two things that we can compare these calls to. The first is the proposed performance hint API. However, this API is just about hints and may not be a good enough analogy for the rest of the forum. The other analogy is the performance profiling APIs that some MPI implementation support. These APIs allow tools to determine some statistics about internal MPI state. If that is the analogy that is drawn, then it is bad for this proposal because I don't think that the performance profiling API ever got much support because of the issues that we're discussing here. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 02:03 PM 10/24/2008, Supalov, Alexander wrote: >Thanks. I can't speak for the whole Forum, but my impression is that if >the choice will be between solving the problem of MPI and CR on one >hand, and not solving it on the other hand, a reasonable proposal will >go a long way toward convincing the majority, or at least moving the >discussion to a still better proposal. > >As for the number of calls, this is question of ROI. We're going to add >200 or so fancy calls by the latest guess, while here we have just 2 >that offer basic functionality of undeniable value. This should be >acceptable. >Finally, I don't know a more implementation specific call than MPI_Init. >The proposed calls live close nearby.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 24 Oct '08

24 Oct '08

Thanks. I can't speak for the whole Forum, but my impression is that if the choice will be between solving the problem of MPI and CR on one hand, and not solving it on the other hand, a reasonable proposal will go a long way toward convincing the majority, or at least moving the discussion to a still better proposal. As for the number of calls, this is question of ROI. We're going to add 200 or so fancy calls by the latest guess, while here we have just 2 that offer basic functionality of undeniable value. This should be acceptable. Finally, I don't know a more implementation specific call than MPI_Init. The proposed calls live close nearby. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Friday, October 24, 2008 10:51 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements >Probably we should not try to define this, and encourage >checkpoint/restart people work with particular MPI implementations to >make sure things work? After all, it's not about an abstract MPI and >abstract CR system - there will probably always be actual pairs (or >other relationships) of them that can work together. > >Compare this to the situation with threads. MPI acknowledges their >existence and provide a couple of calls to request a particular level of >support, that's all. This was good enough for starters, and may change >in the future. The relation between MPI and CR may go this way, too: >acknowledging first, integrating next. If all that you want to add to the spec is a couple of calls to say that MPI state should be frozen in some sense, that's fine with me since we're defining what these calls do. However, I'm not sure that the wider forum will accept them because there is a general dislike for adding more calls and the primary value of these calls is to motivate people to provide implementation-specific definitions for them, which I'm guessing is lower than the standard bar for acceptance. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Greg Bronevetsky 24 Oct '08

24 Oct '08

>Probably we should not try to define this, and encourage >checkpoint/restart people work with particular MPI implementations to >make sure things work? After all, it's not about an abstract MPI and >abstract CR system - there will probably always be actual pairs (or >other relationships) of them that can work together. > >Compare this to the situation with threads. MPI acknowledges their >existence and provide a couple of calls to request a particular level of >support, that's all. This was good enough for starters, and may change >in the future. The relation between MPI and CR may go this way, too: >acknowledging first, integrating next. If all that you want to add to the spec is a couple of calls to say that MPI state should be frozen in some sense, that's fine with me since we're defining what these calls do. However, I'm not sure that the wider forum will accept them because there is a general dislike for adding more calls and the primary value of these calls is to motivate people to provide implementation-specific definitions for them, which I'm guessing is lower than the standard bar for acceptance. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 24 Oct '08

24 Oct '08

Probably we should not try to define this, and encourage checkpoint/restart people work with particular MPI implementations to make sure things work? After all, it's not about an abstract MPI and abstract CR system - there will probably always be actual pairs (or other relationships) of them that can work together. Compare this to the situation with threads. MPI acknowledges their existence and provide a couple of calls to request a particular level of support, that's all. This was good enough for starters, and may change in the future. The relation between MPI and CR may go this way, too: acknowledging first, integrating next. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Friday, October 24, 2008 10:38 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements My problem is that while this has clean semantics for an application-level checkpointer, the same is not true for a system-level checkpointer. In the latter case the checkpointer can exactly capture an undefined subset of MPI state (ex: main memory state but not network card state). As such, MPI_PREPARE_FOR_CHECKPOINT would essentially tell MPI to pull all of its state into the subset that can be checkpointed by the system-level checkpointer. However, the subset depends closely on the type of checkpointing being performed, which itself is very system-specific. As such, I don't know how to provide semantics for this call without using low-level language. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 01:32 PM 10/24/2008, Supalov, Alexander wrote: >Hi everybody, > >I'm afraid we're overcomplicating things a little here. What we need are >basically two collective calls: > >MPI_PREPARE_FOR_CHECKPOINT >MPI_RESTART_AFTER_CHECKPOINT > >The former is (almost) like MPI_Finalize, the latter is (almost) like >MPI_Init. What they mean is up to the implementation, with one >condition: it must be possible to do actual checkpoint/restart in >between. > >I cannot exclude that the exact meaning of the calls and the notice will >be influenced by implementation details like memory registration, the >checkpoint/restart system used, the network involved, etc. > >These collective calls may be complemented by individual, non-collective >calls if needed. They will be suitable for individual >checkpoint/restart, and the user will have to make sure no bad things >happen, like messages trying to reach a process, the memory of which is >currently being dumped. > >Best regards. > >Alexander > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg >Bronevetsky >Sent: Friday, October 24, 2008 10:21 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] system-level C/R requirements > > > >Agreed -- specifiying an explicit list of platforms or OS or even > >resource specifics is not the way to go in a standard. > > > >My suggestion would be to explore if we can define abstract, > >higher-level resources to define a "state", and specify high-level > >actions. For instance, pinning/unpinning memory is very specific to > >RDMA, but maybe a "disconnect virtual connection" operation may > >abstract it. But this puts us into the realm of virtualizing MPI > >internal components/concepts .. > > > >Maybe there is a more elegant way ... >The thing that worries me is that an MPI implementation may have a >fair amount of state sitting on the network card. This state is >unreachable by a user- or kernel-level checkpointer but may be >reachable by a VMM-level checkpointer. How do we differentiate the >level at which we're working? System-level checkpointers working at >different levels need MPI state to be flushed to different levels of >abstraction and it seems that we'll need to be very low-level in >order to define what it means to operate at a given level of >abstraction. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 24 Oct '08

24 Oct '08

PPS. Note that in this case the actual checkpoint/restart probably should be done by means outside of the MPI. It's none of our business to tell those guys how to do this and what calls to use - they should figure this out themselves, individually or together. Our job is to make sure what they do will be based on a consistent and restorable program state. I can even imagine that this may be an application level (i.e., homegrown array I/O) rather than system level checkpoint/restart, if the application allows this and MPI goes along. -----Original Message----- From: Supalov, Alexander Sent: Friday, October 24, 2008 10:34 PM To: 'MPI 3.0 Fault Tolerance and Dynamic Process Control working Group' Subject: RE: [Mpi3-ft] system-level C/R requirements PS. Better still, add a communicator argument to these calls and be happy: both extremes as well as anything in between will be covered. How much will be supported is again up to the implementation. -----Original Message----- From: Supalov, Alexander Sent: Friday, October 24, 2008 10:32 PM To: 'MPI 3.0 Fault Tolerance and Dynamic Process Control working Group' Subject: RE: [Mpi3-ft] system-level C/R requirements Hi everybody, I'm afraid we're overcomplicating things a little here. What we need are basically two collective calls: MPI_PREPARE_FOR_CHECKPOINT MPI_RESTART_AFTER_CHECKPOINT The former is (almost) like MPI_Finalize, the latter is (almost) like MPI_Init. What they mean is up to the implementation, with one condition: it must be possible to do actual checkpoint/restart in between. I cannot exclude that the exact meaning of the calls and the notice will be influenced by implementation details like memory registration, the checkpoint/restart system used, the network involved, etc. These collective calls may be complemented by individual, non-collective calls if needed. They will be suitable for individual checkpoint/restart, and the user will have to make sure no bad things happen, like messages trying to reach a process, the memory of which is currently being dumped. Best regards. Alexander -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Friday, October 24, 2008 10:21 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements >Agreed -- specifiying an explicit list of platforms or OS or even >resource specifics is not the way to go in a standard. > >My suggestion would be to explore if we can define abstract, >higher-level resources to define a "state", and specify high-level >actions. For instance, pinning/unpinning memory is very specific to >RDMA, but maybe a "disconnect virtual connection" operation may >abstract it. But this puts us into the realm of virtualizing MPI >internal components/concepts .. > >Maybe there is a more elegant way ... The thing that worries me is that an MPI implementation may have a fair amount of state sitting on the network card. This state is unreachable by a user- or kernel-level checkpointer but may be reachable by a VMM-level checkpointer. How do we differentiate the level at which we're working? System-level checkpointers working at different levels need MPI state to be flushed to different levels of abstraction and it seems that we'll need to be very low-level in order to define what it means to operate at a given level of abstraction. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Greg Bronevetsky 24 Oct '08

24 Oct '08

My problem is that while this has clean semantics for an application-level checkpointer, the same is not true for a system-level checkpointer. In the latter case the checkpointer can exactly capture an undefined subset of MPI state (ex: main memory state but not network card state). As such, MPI_PREPARE_FOR_CHECKPOINT would essentially tell MPI to pull all of its state into the subset that can be checkpointed by the system-level checkpointer. However, the subset depends closely on the type of checkpointing being performed, which itself is very system-specific. As such, I don't know how to provide semantics for this call without using low-level language. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 01:32 PM 10/24/2008, Supalov, Alexander wrote: >Hi everybody, > >I'm afraid we're overcomplicating things a little here. What we need are >basically two collective calls: > >MPI_PREPARE_FOR_CHECKPOINT >MPI_RESTART_AFTER_CHECKPOINT > >The former is (almost) like MPI_Finalize, the latter is (almost) like >MPI_Init. What they mean is up to the implementation, with one >condition: it must be possible to do actual checkpoint/restart in >between. > >I cannot exclude that the exact meaning of the calls and the notice will >be influenced by implementation details like memory registration, the >checkpoint/restart system used, the network involved, etc. > >These collective calls may be complemented by individual, non-collective >calls if needed. They will be suitable for individual >checkpoint/restart, and the user will have to make sure no bad things >happen, like messages trying to reach a process, the memory of which is >currently being dumped. > >Best regards. > >Alexander > >-----Original Message----- >From: mpi3-ft-bounces(a)lists.mpi-forum.org >[mailto:[email protected]] On Behalf Of Greg >Bronevetsky >Sent: Friday, October 24, 2008 10:21 PM >To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; >MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [Mpi3-ft] system-level C/R requirements > > > >Agreed -- specifiying an explicit list of platforms or OS or even > >resource specifics is not the way to go in a standard. > > > >My suggestion would be to explore if we can define abstract, > >higher-level resources to define a "state", and specify high-level > >actions. For instance, pinning/unpinning memory is very specific to > >RDMA, but maybe a "disconnect virtual connection" operation may > >abstract it. But this puts us into the realm of virtualizing MPI > >internal components/concepts .. > > > >Maybe there is a more elegant way ... >The thing that worries me is that an MPI implementation may have a >fair amount of state sitting on the network card. This state is >unreachable by a user- or kernel-level checkpointer but may be >reachable by a VMM-level checkpointer. How do we differentiate the >level at which we're working? System-level checkpointers working at >different levels need MPI state to be flushed to different levels of >abstraction and it seems that we'll need to be very low-level in >order to define what it means to operate at a given level of >abstraction. > >Greg Bronevetsky >Post-Doctoral Researcher >1028 Building 451 >Lawrence Livermore National Lab >(925) 424-5756 >bronevetsky1(a)llnl.gov > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >--------------------------------------------------------------------- >Intel GmbH >Dornacher Strasse 1 >85622 Feldkirchen/Muenchen Germany >Sitz der Gesellschaft: Feldkirchen bei Muenchen >Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer >Registergericht: Muenchen HRB 47456 Ust.-IdNr. >VAT Registration No.: DE129385895 >Citibank Frankfurt (BLZ 502 109 00) 600119052 > >This e-mail and any attachments may contain confidential material for >the sole use of the intended recipient(s). Any review or distribution >by others is strictly prohibited. If you are not the intended >recipient, please contact the sender and delete all copies. > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 24 Oct '08

24 Oct '08

PS. Better still, add a communicator argument to these calls and be happy: both extremes as well as anything in between will be covered. How much will be supported is again up to the implementation. -----Original Message----- From: Supalov, Alexander Sent: Friday, October 24, 2008 10:32 PM To: 'MPI 3.0 Fault Tolerance and Dynamic Process Control working Group' Subject: RE: [Mpi3-ft] system-level C/R requirements Hi everybody, I'm afraid we're overcomplicating things a little here. What we need are basically two collective calls: MPI_PREPARE_FOR_CHECKPOINT MPI_RESTART_AFTER_CHECKPOINT The former is (almost) like MPI_Finalize, the latter is (almost) like MPI_Init. What they mean is up to the implementation, with one condition: it must be possible to do actual checkpoint/restart in between. I cannot exclude that the exact meaning of the calls and the notice will be influenced by implementation details like memory registration, the checkpoint/restart system used, the network involved, etc. These collective calls may be complemented by individual, non-collective calls if needed. They will be suitable for individual checkpoint/restart, and the user will have to make sure no bad things happen, like messages trying to reach a process, the memory of which is currently being dumped. Best regards. Alexander -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Friday, October 24, 2008 10:21 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements >Agreed -- specifiying an explicit list of platforms or OS or even >resource specifics is not the way to go in a standard. > >My suggestion would be to explore if we can define abstract, >higher-level resources to define a "state", and specify high-level >actions. For instance, pinning/unpinning memory is very specific to >RDMA, but maybe a "disconnect virtual connection" operation may >abstract it. But this puts us into the realm of virtualizing MPI >internal components/concepts .. > >Maybe there is a more elegant way ... The thing that worries me is that an MPI implementation may have a fair amount of state sitting on the network card. This state is unreachable by a user- or kernel-level checkpointer but may be reachable by a VMM-level checkpointer. How do we differentiate the level at which we're working? System-level checkpointers working at different levels need MPI state to be flushed to different levels of abstraction and it seems that we'll need to be very low-level in order to define what it means to operate at a given level of abstraction. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Supalov, Alexander 24 Oct '08

24 Oct '08

Hi everybody, I'm afraid we're overcomplicating things a little here. What we need are basically two collective calls: MPI_PREPARE_FOR_CHECKPOINT MPI_RESTART_AFTER_CHECKPOINT The former is (almost) like MPI_Finalize, the latter is (almost) like MPI_Init. What they mean is up to the implementation, with one condition: it must be possible to do actual checkpoint/restart in between. I cannot exclude that the exact meaning of the calls and the notice will be influenced by implementation details like memory registration, the checkpoint/restart system used, the network involved, etc. These collective calls may be complemented by individual, non-collective calls if needed. They will be suitable for individual checkpoint/restart, and the user will have to make sure no bad things happen, like messages trying to reach a process, the memory of which is currently being dumped. Best regards. Alexander -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Greg Bronevetsky Sent: Friday, October 24, 2008 10:21 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] system-level C/R requirements >Agreed -- specifiying an explicit list of platforms or OS or even >resource specifics is not the way to go in a standard. > >My suggestion would be to explore if we can define abstract, >higher-level resources to define a "state", and specify high-level >actions. For instance, pinning/unpinning memory is very specific to >RDMA, but maybe a "disconnect virtual connection" operation may >abstract it. But this puts us into the realm of virtualizing MPI >internal components/concepts .. > >Maybe there is a more elegant way ... The thing that worries me is that an MPI implementation may have a fair amount of state sitting on the network card. This state is unreachable by a user- or kernel-level checkpointer but may be reachable by a VMM-level checkpointer. How do we differentiate the level at which we're working? System-level checkpointers working at different levels need MPI state to be flushed to different levels of abstraction and it seems that we'll need to be very low-level in order to define what it means to operate at a given level of abstraction. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] system-level C/R requirements
by Greg Bronevetsky 24 Oct '08

24 Oct '08

>Agreed -- specifiying an explicit list of platforms or OS or even >resource specifics is not the way to go in a standard. > >My suggestion would be to explore if we can define abstract, >higher-level resources to define a "state", and specify high-level >actions. For instance, pinning/unpinning memory is very specific to >RDMA, but maybe a "disconnect virtual connection" operation may >abstract it. But this puts us into the realm of virtualizing MPI >internal components/concepts .. > >Maybe there is a more elegant way ... The thing that worries me is that an MPI implementation may have a fair amount of state sitting on the network card. This state is unreachable by a user- or kernel-level checkpointer but may be reachable by a VMM-level checkpointer. How do we differentiate the level at which we're working? System-level checkpointers working at different levels need MPI state to be flushed to different levels of abstraction and it seems that we'll need to be very low-level in order to define what it means to operate at a given level of abstraction. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0