October 2008 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Summary of today's meeting
by Thomas Herault 21 Oct '08

21 Oct '08

Le 21 oct. 08 ï¿½ 22:06, Howard Pritchard a ï¿½crit : > Hello Rich, > > I thought it was also agreed that if process A communicates with > failed process B > which had been restarted by another process C, and this was the > first communication > from A to B since the restart of B, A would receive the equivalent > of a ECONNRESET error. > This was in the context of a case where option 5 below is not being > used by the app. > > Howard > Hello Howard, there was still some discussions about this at the end of the session. The argument is that the application could do as well as the library to enforce this detection if this is needed: when a process is launched to replace another one, it could define a new revision/epoch/ restart number and tag each communication with this number to implement the check. If this can be done as efficiently by the application as it would be done by the library, asking the application to do it itself would help the library to avoid the additional cost (i.e. piggybacking an integer to each message) when the application does not need that functionality. It was suggested that the library could provide a generic mean to piggyback this kind of information to each message, in a way similar as what is discussed about piggyback/message logging-based fault tolerance. Thomas > Richard Graham wrote: >> >> Here is a summary of what I think that we agreed to today. Please >> correct any errors, and add what I am missing. >> >> ï¿½ We need to be able to restore MPI_COMM_WORLD (and itï¿½s >> derivatives) to a usable state when a process fails. >> ï¿½ Restoration may involve having MPI_PROC_NULL replace the lost >> process, or may replaced the lost processes with a new process >> (have not specified how this would happen) >> ï¿½ Processes communicating directly with the failed processes will >> be notified via a returned error code about the failure. >> ï¿½ When a process is notified of the failure, comm_repair() must be >> called. Comm_repair() is not a collective call, and is what will >> initiate the communicator repair associated with the failed process. >> ï¿½ If a process wants to be notified of process failure even if it >> is not communicating directly with this process, it must register >> for this notification. >> ï¿½ We donï¿½t have enough information to know how to continue with >> support for checkpoint/restart. >> ï¿½ We need to discuss what needs to do with respect to failure of >> collective communications. >> >> There are several issues that came up with respect to these, which >> will be detailed later on. >> >> Rich >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > -- > > Howard Pritchard > Cray Inc. > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Summary of today's meeting
by Howard Pritchard 21 Oct '08

21 Oct '08

1 0

[Mpi3-ft] Meeting at SC
by Richard Graham 21 Oct '08

21 Oct '08

We talked today about meeting at SC to continue making progress on the FT work. I would like to have a 3 hour chunk of time for this, if we can get people together for this long. Here are some proposed times: Tuesday anytime in the 9am 1:30 pm time frame. Wed before noon Thursday Anytime but 1:30 3:00 pm. What is a good time for people ? Does someone have a location where we can meet ? Thanks, Rich

1 0

[Mpi3-ft] Summary of today's meeting
by Richard Graham 21 Oct '08

21 Oct '08

Here is a summary of what I think that we agreed to today. Please correct any errors, and add what I am missing. * We need to be able to restore MPI_COMM_WORLD (and it¹s derivatives) to a usable state when a process fails. * Restoration may involve having MPI_PROC_NULL replace the lost process, or may replaced the lost processes with a new process (have not specified how this would happen) * Processes communicating directly with the failed processes will be notified via a returned error code about the failure. * When a process is notified of the failure, comm_repair() must be called. Comm_repair() is not a collective call, and is what will initiate the communicator repair associated with the failed process. * If a process wants to be notified of process failure even if it is not communicating directly with this process, it must register for this notification. * We don¹t have enough information to know how to continue with support for checkpoint/restart. * We need to discuss what needs to do with respect to failure of collective communications. There are several issues that came up with respect to these, which will be detailed later on. Rich

1 0

Re: [Mpi3-ft] Con call this week
by Nathan DeBardeleben 14 Oct '08

14 Oct '08

I'm at LACSS this week so I'll miss the telecon. I'll try and put up the LANL use cases we discussed on the last telecon today. -- Nathan --------------------------------------------------------------------- Nathan DeBardeleben, Ph.D. Los Alamos National Laboratory High Performance Computing Systems Integration (HPC-5) phone: 505-667-3428 email: ndebard(a)lanl.gov --------------------------------------------------------------------- Richard Graham wrote: > Just a reminder that we will have our regularly scheduled call this > coming Wed. Call details are: > > Wed - 12-2 pm EDT, 5-7 pm GMT > US Toll Free number: 877-801-8130 > Toll number: 1-203-692-8690 > Access Code: 1044056 > > I think that we are close to competing the discussion of specific use > cases, so lets aim to close these off on this call. Please remember > to add your use cases to the wiki, so that we can consider these for > the next round. Noticeably missing contributions are from those > interested in building checkpointing libraries on top of a > fault-tolerant MPI (i.e., w/o having to change the MPI implementation) > - if anyone has contributions along these lines, please speak up. We > have talked about these in the past, but now we need to consolidate > the various inputs, so we can proceed in a coherent manner. > > At the forum meeting in a week I would like to have us start working > on semantics and an API for support for process fault-tolerance, with > the goal to be in position to start prototyping implementations after > the Dec. meeting. > > Rich > ------------------------------------------------------------------------ > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

[Mpi3-ft] Con call this week
by Richard Graham 11 Oct '08

11 Oct '08

Just a reminder that we will have our regularly scheduled call this coming Wed. Call details are: Wed - 12-2 pm EDT, 5-7 pm GMT US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 I think that we are close to competing the discussion of specific use cases, so lets aim to close these off on this call. Please remember to add your use cases to the wiki, so that we can consider these for the next round. Noticeably missing contributions are from those interested in building checkpointing libraries on top of a fault-tolerant MPI (i.e., w/o having to change the MPI implementation) - if anyone has contributions along these lines, please speak up. We have talked about these in the past, but now we need to consolidate the various inputs, so we can proceed in a coherent manner. At the forum meeting in a week I would like to have us start working on semantics and an API for support for process fault-tolerance, with the goal to be in position to start prototyping implementations after the Dec. meeting. Rich

1 0

Re: [Mpi3-ft] FT WG Meeting today?
by Erez Haba 08 Oct '08

08 Oct '08

I think that last week Rich said we will not have a meeting today. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Narasimhan, Kannan Sent: Wednesday, October 08, 2008 9:08 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] FT WG Meeting today? Are we having the FT Working Group meeting today? Looping on music .... Thanx! Kannan _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] Reminder: no call today
by Richard Graham 08 Oct '08

08 Oct '08

Just a reminder that we decided last week we would not have a call today we will resume next week. Rich

1 0

[Mpi3-ft] FT WG Meeting today?
by Narasimhan, Kannan 08 Oct '08

08 Oct '08

Are we having the FT Working Group meeting today? Looping on music .... Thanx! Kannan

1 0