January 2008 - mpiwg-ft - mailman3.cels.anl.gov

[mpi3-ft] Notes from 15 Jan. 2008 Meeting
by Josh Hursey 22 Jan '08

22 Jan '08

Last week Rich asked me to take some notes from the MPI Forum discussion on dynamic process/fault tolerance. Those notes are enclosed. --Josh ------------------- MPI Forum 3.0 Dynamic Process/Fault Tolerance Working Group Brief Notes from Jan. 15, 2008 Meeting - Chicago, IL Chapter Coordinator: Rich Graham Notes taken by Josh Hursey * May have to describe the state of a function across a failure (and possible recovery) for all MPI functions and failure scenarios * May want to consider proposing a suggestion for implementation: - i.e.: An MPI implementation does not need to do X, but if it does decide to support X then we recommend the following as a way of implementing it. * Provide support for some class (or subset) of recovery mechanisms that may be able to be built on top of an MPI implementation of the MPI 3.0 standard. * Replication techniques should be added to the (presented) list of possible recovery techniques to consider supporting. - Question: Does considering replication techniques undermine the HPC target community of the MPI standard. - MPI interface should allow for this, maybe as an opt'in functionality. * An interface to easily piggyback data on point-to-point (and maybe collective) messages would help support some high level checkpoint/ restart techniques. * Should we expose to the application the fault recovery mechanism used? (i.e., Network path failover) - As a notification once complete? - Or as a request to authorize the recovery mechanism before it executes? * Some users have expressed an interest in having an explicit checkpoint function to identify 'good' times in the code execution to checkpoint. Ideally when the checkpoint state is minimal. * Can we ask the MPI to save some internal state around a user level checkpoint operation such that on recovery MPI objects such as datatypes or communicators are automatically recreated for the user before the application resumes execution. * Areas of MPI possibly affected by fault tolerance: - Communicators such as MPI_COMM_WORLD may change size. - MPI_Init/MPI_Finalize may need to called multiple times to support some recovery techniques. - MPI I/O needs to be looked at. - MPI Topology changes that may/will occur when a process failure and possible recovery occurs. * Collective communication: - For some collectives a process can be in the collective or on either side of it, how do the semantics of the collective operation change in response to a failure/recovery operation. - Bcast, for example, may allow some processes to exit early providing some performance benefits. A fault tolerance semantic for Bcast may require a global synchronization at the end of the Bcast to ensure a two-phase commit of the effected buffers. - We could consider loosening the synchronization constraints such that the user could choose, via an MPI API, the level of confidence they require for a process to exit a collective call. - Maybe introduce the notion of epochs that can be started and ended to help in fault isolation. * Use case: How can a user choose to enable or disable fault tolerance features? - Mostly implementation specific. (link time, compile time, ...) * Could fault tolerance only apply to a subset of the MPI API for simplicity? For example, exclude collective communications or MPI I/O. * It was stated that we should express that the fault tolerance MPI specification may not be able to provide guarantees about the stability of some data segments or operations effected by a fault, but that an MPI implementation will provide a stable enough environment to allow for post recovery. - This restriction should be explicitly stated in the standard text presented. * In a failure scenario is the fault mode/error reported local or global to the communicator? * Most people agree that this working group should step forward towards API changes to help support, but maybe not provide, fault tolerance. * A mailing list will be created for this group, and will be posted to the mpi-21 listserv.

1 0

Re: [mpi3-ft] Picking up working group activities
by Supalov, Alexander 22 Jan '08

22 Jan '08

Hi, 11am EST Wednesday is OK with me. I think of faults as asynchronous errors. They can be reported during the next nearest MPI call or asynchronously. Like in the case of thread support, the user should be able to select what way he wants his/her fault notifications served. Likewise, the application should be able to provide some maximum support level (say, MPI_FAULT_SYNCHRONOUS, MPI_FAULT_ASYNCHRONOUS). The handling of the faults should be analogous to the handling of the usual MPI errors. The mechanism of error handlers seems adequate. We'd probably have all faults fatal by default (MPI_FAULTS_ARE_FATAL). If the user wants to do something more intelligent about them, another handler should be defined and registered with the MPI. We may wish to provide some predefined handlers, like MPI_FAULTS_ARE_IGNORED. Finally, fault notifications sent to other processes. Again, there should be several modes and both the user and the implementation should be able to select what they want to use or can provide, respectively. Notifications may go to all processes of the job, all processes in the affected communicator, only the root of it, or no process but the affected one. Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 -----Original Message----- From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Monday, January 21, 2008 9:28 PM To: Discussion of MPI 3 Fault Tolerance Support Subject: Re: [mpi3-ft] Picking up working group activities I'm in US Eastern time zone. Wednesday is fine with me except between 11:30 am - 1 pm EST. Thursday is completely clear for me, as is Tuesday afternoon EST. -- Josh On Jan 21, 2008, at 2:45 PM, Greg Bronevetsky wrote: > At 10:14 AM 1/21/2008, Richard Graham wrote: >> I would like to start bi-weekly con calls to discuss Fault >> Tolerance and >> dynamic process support in the context of MPI 3.0. First, we need >> to find >> a time for the telecon that works for most people, so I will start by >> suggesting that we have the call on Wed's at 11 am EST, starting >> 1/30/2008. >> How does this work for people who plan to be active participants in >> this >> work ? > > I can't do 11amEST on Wednesday because that is the OpenMP conference > call. As others have pointed out, the time was chosen to fit in both > Europe and the US. There is no combination that I know of that is > convenient for US, Europe and Asia, so maybe we should first poll > people's time zones to pick the best time zone compromise. > > I'm in the US Pacific time zone. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] Picking up working group activities
by Josh Hursey 21 Jan '08

21 Jan '08

I'm in US Eastern time zone. Wednesday is fine with me except between 11:30 am - 1 pm EST. Thursday is completely clear for me, as is Tuesday afternoon EST. -- Josh On Jan 21, 2008, at 2:45 PM, Greg Bronevetsky wrote: > At 10:14 AM 1/21/2008, Richard Graham wrote: >> I would like to start bi-weekly con calls to discuss Fault >> Tolerance and >> dynamic process support in the context of MPI 3.0. First, we need >> to find >> a time for the telecon that works for most people, so I will start by >> suggesting that we have the call on Wed's at 11 am EST, starting >> 1/30/2008. >> How does this work for people who plan to be active participants in >> this >> work ? > > I can't do 11amEST on Wednesday because that is the OpenMP conference > call. As others have pointed out, the time was chosen to fit in both > Europe and the US. There is no combination that I know of that is > convenient for US, Europe and Asia, so maybe we should first poll > people's time zones to pick the best time zone compromise. > > I'm in the US Pacific time zone. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

Re: [mpi3-ft] Picking up working group activities
by Greg Bronevetsky 21 Jan '08

21 Jan '08

At 10:14 AM 1/21/2008, Richard Graham wrote: >I would like to start bi-weekly con calls to discuss Fault Tolerance and >dynamic process support in the context of MPI 3.0. First, we need to find >a time for the telecon that works for most people, so I will start by >suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. >How does this work for people who plan to be active participants in this >work ? I can't do 11amEST on Wednesday because that is the OpenMP conference call. As others have pointed out, the time was chosen to fit in both Europe and the US. There is no combination that I know of that is convenient for US, Europe and Asia, so maybe we should first poll people's time zones to pick the best time zone compromise. I'm in the US Pacific time zone. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [mpi3-ft] Picking up working group activities
by Pavan Balaji 21 Jan '08

21 Jan '08

Rich, 11a EST or later will be fine with me. Here are some things that we might want to discuss, depending on interest, during the telecon: 1. Level of fault tolerance --- should the MPI implementation provide information about what level of fault tolerance it supports (similar to threads). Choices could be: (a) None, (b) MPI provides error description and allows application to save information and abort (cannot continue), (c) MPI provides error description and allows for recovery in some cases. 2. Turn-off non-transparent fault-tolerance: If the MPI implementation is capable of automatically dealing with faults, the application should be allowed to turn on/off each feature. For example, if the MPI implementation supports auto-migration to a different network when a network fails (or is giving performance problems), the application should be able to shut off this capability if it doesn't want it. 3. Distinguish FAULTS vs. HINTS. Faults can be considered fatal errors, while hints are non-fatal (e.g., performance is not optimal). 4. User directives --- if the MPI implementation supports some capability (e.g., checkpoint), should the application be allowed to force (or request) it to happen at some specific time? Thanks. -- Pavan On 01/21/2008 01:08 PM, Erez Haba wrote: > Thanks Rich, > > Wednesday's are okay for me; however 11am EST is too early. 1pm or 2pm EST would work better for me thought it might not for people from Europe. > >>From my pov, resolving the MPI_COMM_WORD FT behavior is the highest priority issue. > > > -----Original Message----- > From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Richard Graham > Sent: Monday, January 21, 2008 10:15 AM > To: mpi3-ft(a)cs.uiuc.edu > Subject: [mpi3-ft] Picking up working group activities > > I would like to start bi-weekly con calls to discuss Fault Tolerance and > dynamic process support in the context of MPI 3.0. First, we need to find > a time for the telecon that works for most people, so I will start by > suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. > How does this work for people who plan to be active participants in this > work ? > > The items I would like to start addressing next week are > - Does MPI provide FT, or enable FT work ? > - Do we need more than is available in the current standard ? > - If so, what are the use-case scenarios that we are aiming to support ? > > These seem to be a good starting point for discussions. If you have a > particular use-case you want us to consider in this working group, please > write it up and circulate it to on the mailing list at least a day before we > meet, to give people a chance to read these over. Please keep the > descriptions high-level, avoiding specific solutions, at this stage. We > will start to discuss specifics once we have defined the scope of the > problem that needs to be addressed - if we agree that there is one in the > first place. > > Same thing for dynamic process support - if you have specific use-case > scenarios that are not supported under the current standard, and you would > like to have these considered for inclusion in the 3.0 standard, please > provide the use cases you would like the group to consider. > > We should continue to take input on use case scenarios until the next > face-to-face meeting, but the sooner we get these, the more effective we > will be in proceeding in a timely manner. > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpi3-ft] Picking up working group activities
by Yann K. 21 Jan '08

21 Jan '08

Erez noticed the detail. Thanks. If somebody joins and is from Asia, we're trapped ! In Europe, can't do anything if not between 9 and 11am EST. Check this example : http://www.timeanddate.com/worldclock/meetingtime.html?month=1&day=21&year=… Yann Erez Haba wrote: > Thanks Rich, > > Wednesday's are okay for me; however 11am EST is too early. 1pm or 2pm EST would work better for me thought it might not for people from Europe. > > >From my pov, resolving the MPI_COMM_WORD FT behavior is the highest priority issue. > > > -----Original Message----- > From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Richard Graham > Sent: Monday, January 21, 2008 10:15 AM > To: mpi3-ft(a)cs.uiuc.edu > Subject: [mpi3-ft] Picking up working group activities > > I would like to start bi-weekly con calls to discuss Fault Tolerance and > dynamic process support in the context of MPI 3.0. First, we need to find > a time for the telecon that works for most people, so I will start by > suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. > How does this work for people who plan to be active participants in this > work ? > > The items I would like to start addressing next week are > - Does MPI provide FT, or enable FT work ? > - Do we need more than is available in the current standard ? > - If so, what are the use-case scenarios that we are aiming to support ? > > These seem to be a good starting point for discussions. If you have a > particular use-case you want us to consider in this working group, please > write it up and circulate it to on the mailing list at least a day before we > meet, to give people a chance to read these over. Please keep the > descriptions high-level, avoiding specific solutions, at this stage. We > will start to discuss specifics once we have defined the scope of the > problem that needs to be addressed - if we agree that there is one in the > first place. > > Same thing for dynamic process support - if you have specific use-case > scenarios that are not supported under the current standard, and you would > like to have these considered for inclusion in the 3.0 standard, please > provide the use cases you would like the group to consider. > > We should continue to take input on use case scenarios until the next > face-to-face meeting, but the sooner we get these, the more effective we > will be in proceeding in a timely manner. > > Thanks, > Rich > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft > >

1 0

Re: [mpi3-ft] Picking up working group activities
by Erez Haba 21 Jan '08

21 Jan '08

Thanks Rich, Wednesday's are okay for me; however 11am EST is too early. 1pm or 2pm EST would work better for me thought it might not for people from Europe. >From my pov, resolving the MPI_COMM_WORD FT behavior is the highest priority issue. -----Original Message----- From: mpi3-ft-bounces(a)cs.uiuc.edu [mailto:[email protected]] On Behalf Of Richard Graham Sent: Monday, January 21, 2008 10:15 AM To: mpi3-ft(a)cs.uiuc.edu Subject: [mpi3-ft] Picking up working group activities I would like to start bi-weekly con calls to discuss Fault Tolerance and dynamic process support in the context of MPI 3.0. First, we need to find a time for the telecon that works for most people, so I will start by suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. How does this work for people who plan to be active participants in this work ? The items I would like to start addressing next week are - Does MPI provide FT, or enable FT work ? - Do we need more than is available in the current standard ? - If so, what are the use-case scenarios that we are aiming to support ? These seem to be a good starting point for discussions. If you have a particular use-case you want us to consider in this working group, please write it up and circulate it to on the mailing list at least a day before we meet, to give people a chance to read these over. Please keep the descriptions high-level, avoiding specific solutions, at this stage. We will start to discuss specifics once we have defined the scope of the problem that needs to be addressed - if we agree that there is one in the first place. Same thing for dynamic process support - if you have specific use-case scenarios that are not supported under the current standard, and you would like to have these considered for inclusion in the 3.0 standard, please provide the use cases you would like the group to consider. We should continue to take input on use case scenarios until the next face-to-face meeting, but the sooner we get these, the more effective we will be in proceeding in a timely manner. Thanks, Rich _______________________________________________ mpi3-ft mailing list mpi3-ft(a)cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/mpi3-ft

1 0

[mpi3-ft] Picking up working group activities
by Richard Graham 21 Jan '08

21 Jan '08

I would like to start bi-weekly con calls to discuss Fault Tolerance and dynamic process support in the context of MPI 3.0. First, we need to find a time for the telecon that works for most people, so I will start by suggesting that we have the call on Wed's at 11 am EST, starting 1/30/2008. How does this work for people who plan to be active participants in this work ? The items I would like to start addressing next week are - Does MPI provide FT, or enable FT work ? - Do we need more than is available in the current standard ? - If so, what are the use-case scenarios that we are aiming to support ? These seem to be a good starting point for discussions. If you have a particular use-case you want us to consider in this working group, please write it up and circulate it to on the mailing list at least a day before we meet, to give people a chance to read these over. Please keep the descriptions high-level, avoiding specific solutions, at this stage. We will start to discuss specifics once we have defined the scope of the problem that needs to be addressed - if we agree that there is one in the first place. Same thing for dynamic process support - if you have specific use-case scenarios that are not supported under the current standard, and you would like to have these considered for inclusion in the 3.0 standard, please provide the use cases you would like the group to consider. We should continue to take input on use case scenarios until the next face-to-face meeting, but the sooner we get these, the more effective we will be in proceeding in a timely manner. Thanks, Rich

1 0