August 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] <no subject>
by Erez Haba 03 Aug '08

03 Aug '08

Works for me. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Supalov, Alexander Sent: Saturday, August 02, 2008 11:08 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] <no subject> OK with me, thanks. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Friday, August 01, 2008 10:20 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] <no subject> On the call today we agreed to start meeting on a weekly basis to accelerate progress on the FT issues. There was a request to move the call from the current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an hour). Please let me know if it will work for you, or if it will not work for you - I will assume no response as a "am not involved" - so we can see if this works for people. Please send a response by COB Monday, 8/4, so we can schedule next week's (8/8) call. Thanks, Rich _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] <no subject>
by Supalov, Alexander 02 Aug '08

02 Aug '08

OK with me, thanks. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Friday, August 01, 2008 10:20 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] <no subject> On the call today we agreed to start meeting on a weekly basis to accelerate progress on the FT issues. There was a request to move the call from the current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an hour). Please let me know if it will work for you, or if it will not work for you - I will assume no response as a "am not involved" - so we can see if this works for people. Please send a response by COB Monday, 8/4, so we can schedule next week's (8/8) call. Thanks, Rich _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] A need to detect any failure
by Greg Bronevetsky 01 Aug '08

01 Aug '08

What we're focusing on right now is the most basic notification mechanism that needs to be supported by MPI. This is the notification mechanism that will be used with legacy applications that use error codes to be informed of errors and need sensible default behavior from MPI. Although your use-case makes sense, the extra cost that it would incur means that applications that need this capability are best off using a more complex/customizable API, such as the publish-subscribe API we've been tossing around. Also, the idea of passing error notifications on top of existing messages is good motivation for the piggybacking API. In any case, I think that this is a use-case for the extended notification API, not the default notification mechanism. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov At 11:03 AM 8/1/2008, you wrote: >Hi all, >I like Erez's idea to associate errors with call sites. However, I >still believe there is a case where processes do not communicate >with each other directly, but will need to know when any process in >the system dies. Here is an example. > >Let's say an application splits MPI_COMM_WORLD into a 2d cartesian >grid. Then each process creates a "row" communicator and a "column" >communicator. From here on out, each process only communicates >through its row and column communicators. (I think this is what a >number of our codes really do, so this is a very realistic >example). In the case of a failure, assume this application >requires a global rollback, and assume that each process writes a >checkpoint file at the end of each iteration, which looks like the following: > >for (iteration=0; iteration<numTimesteps; iteration++) { > row_xchange(); > column_xchange(); > compute(); > checkpoint(iteration); >} > >Now consider processes (i, j) and (i+1, j+1) in this 2d cartesion grid. >Because these two processes are in different rows and columns, they >don't share a row or column communicator, and so they never >communicate with each other directly. Now, assume process (i+1, j+1) fails. >Process (i, j) needs to rollback, but how will it be notified? > >One solution would be to force the application to call >MPI_Barrier(MPI_COMM_WORLD) inside of its iteration loop. While >this would work, it seems costly, and it defeats all the effort the >application team went to in order to make the code scalable by using >just row and column communicators. > >Another solution may be to rely on daisy chaing. That is, in the >next iteration after process (i+1, j+1) dies, processes (i+1, j) and >(i, j+1) may find out since they each share a communicator with the >failed process. Then in the following iteration, processes (i+1, j) >and (i, j+1) could propogate this failure message to process (i, j) >since they each share a communicator. This would also work, but a >special error code would be needed since the communication with >(i+1, j) and (i, j+1) may have succeeded just fine. > >In the current standard, MPI handles this type of failure by >invoking the error handler on MPI_COMM_WORLD. This could be yet >another solution. >-Adam >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] <no subject>
by Greg Bronevetsky 01 Aug '08

01 Aug '08

At 01:19 PM 8/1/2008, Richard Graham wrote: >On the call today we agreed to start meeting on a weekly basis to accelerate >progress on the FT issues. There was a request to move the call from the >current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an >hour). Please let me know if it will work for you, or if it will not work >for you - I will assume no response as a "am not involved" - so we can see >if this works for people. Please send a response by COB Monday, 8/4, so we >can schedule next week's (8/8) call. Either time works for me. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] <no subject>
by Hideyuki Jitsumoto 01 Aug '08

01 Aug '08

Richard, This is not good for me. Because, in Japan, new time slot is 2-4 am. Probabry I'll join this meeting every other week. Although, it is no problem, if I can checked menites of meeting... Thank you, Hideyuki Jitsumoto On Sat, Aug 2, 2008 at 5:19 AM, Richard Graham <rlgraham(a)ornl.gov> wrote: > On the call today we agreed to start meeting on a weekly basis to accelerate > progress on the FT issues. There was a request to move the call from the > current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an > hour). Please let me know if it will work for you, or if it will not work > for you - I will assume no response as a "am not involved" - so we can see > if this works for people. Please send a response by COB Monday, 8/4, so we > can schedule next week's (8/8) call. > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Sincerely Yours, Hideyuki Jitsumoto (jitumoto(a)matsulab.is.titech.ac.jp) Tokyo Institute of Technology Global Scientific Information and Computing center (Matsuoka Lab.)

1 0

Re: [Mpi3-ft] <no subject>
by Narasimhan, Kannan 01 Aug '08

01 Aug '08

1 EDT works for me - though I may have to bail out around 2:30 EDT. -Kannan- -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Friday, August 01, 2008 3:20 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] <no subject> On the call today we agreed to start meeting on a weekly basis to accelerate progress on the FT issues. There was a request to move the call from the current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an hour). Please let me know if it will work for you, or if it will not work for you - I will assume no response as a "am not involved" - so we can see if this works for people. Please send a response by COB Monday, 8/4, so we can schedule next week's (8/8) call. Thanks, Rich _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] <no subject>
by Terry Dontje 01 Aug '08

01 Aug '08

Richard Graham wrote: > On the call today we agreed to start meeting on a weekly basis to accelerate > progress on the FT issues. There was a request to move the call from the > current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an > hour). Please let me know if it will work for you, or if it will not work > for you - I will assume no response as a "am not involved" - so we can see > if this works for people. Please send a response by COB Monday, 8/4, so we > can schedule next week's (8/8) call. > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > I have a 2pm ET meeting so I'd prefer the 12-2 ET time period. That being said if there is an overwhelming need to change this to 1-3pm ET than I would be willing to have that happen. --td

1 0

Re: [Mpi3-ft] <no subject>
by Martin Schulz 01 Aug '08

01 Aug '08

Weekly at the 1-3 (actually more likely or 1-2) EDT timeslot would work for. If we keep it at the old time slot I will have to miss every other week. Thanks, Martin At 01:19 PM 8/1/2008, Richard Graham wrote: >On the call today we agreed to start meeting on a weekly basis to accelerate >progress on the FT issues. There was a request to move the call from the >current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an >hour). Please let me know if it will work for you, or if it will not work >for you - I will assume no response as a "am not involved" - so we can see >if this works for people. Please send a response by COB Monday, 8/4, so we >can schedule next week's (8/8) call. > >Thanks, >Rich > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-ft] <no subject>
by Richard Graham 01 Aug '08

01 Aug '08

On the call today we agreed to start meeting on a weekly basis to accelerate progress on the FT issues. There was a request to move the call from the current time of 12-2 EDT to 1-3 EDT (actually, it usually lasts for only an hour). Please let me know if it will work for you, or if it will not work for you - I will assume no response as a "am not involved" - so we can see if this works for people. Please send a response by COB Monday, 8/4, so we can schedule next week's (8/8) call. Thanks, Rich

1 0

[Mpi3-ft] A need to detect any failure
by Adam Moody 01 Aug '08

01 Aug '08

Hi all, I like Erez's idea to associate errors with call sites. However, I still believe there is a case where processes do not communicate with each other directly, but will need to know when any process in the system dies. Here is an example. Let's say an application splits MPI_COMM_WORLD into a 2d cartesian grid. Then each process creates a "row" communicator and a "column" communicator. From here on out, each process only communicates through its row and column communicators. (I think this is what a number of our codes really do, so this is a very realistic example). In the case of a failure, assume this application requires a global rollback, and assume that each process writes a checkpoint file at the end of each iteration, which looks like the following: for (iteration=0; iteration<numTimesteps; iteration++) { row_xchange(); column_xchange(); compute(); checkpoint(iteration); } Now consider processes (i, j) and (i+1, j+1) in this 2d cartesion grid. Because these two processes are in different rows and columns, they don't share a row or column communicator, and so they never communicate with each other directly. Now, assume process (i+1, j+1) fails. Process (i, j) needs to rollback, but how will it be notified? One solution would be to force the application to call MPI_Barrier(MPI_COMM_WORLD) inside of its iteration loop. While this would work, it seems costly, and it defeats all the effort the application team went to in order to make the code scalable by using just row and column communicators. Another solution may be to rely on daisy chaing. That is, in the next iteration after process (i+1, j+1) dies, processes (i+1, j) and (i, j+1) may find out since they each share a communicator with the failed process. Then in the following iteration, processes (i+1, j) and (i, j+1) could propogate this failure message to process (i, j) since they each share a communicator. This would also work, but a special error code would be needed since the communication with (i+1, j) and (i, j+1) may have succeeded just fine. In the current standard, MPI handles this type of failure by invoking the error handler on MPI_COMM_WORLD. This could be yet another solution. -Adam

1 0