March 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] MPI_Comm_validate parameters
by Joshua Hursey 03 Mar '11

03 Mar '11

If we allow the "L_i" and "G" lists to be only updated explicitly by the user, then they are in control over the management of the view of these vectors among their various threads. Per Darius's comment about not allowing MPI_Send to update "L_i", I think this reinforces this constraint though introduces a slightly awkward semantic. I'll take another look at the old documents and see if I can pull out any thread safety conversations. Admittedly, on my initial pass, I was not looking at the thread safety aspects of the original proposal in too much detail so I may have overlooked something subtle. -- Josh On Mar 2, 2011, at 11:49 AM, Graham, Richard L. wrote: > I believe that what is being proposed here is not thread-safe. If you > look back at some of the docs that were put together when we were actively > looking at this over a year ago, there was a proposal on how to do this in > a thread safe manner. Having said that, I am not sure if thread safety is > an issue here or not - it depends very much on the life-cycle of the state > information. In the original proposal, the "vector" returned by > comm_validate had some state information. > > Rich > > On 3/2/11 9:51 AM, "Joshua Hursey" <jjhursey(a)open-mpi.org> wrote: > >> >> On Mar 1, 2011, at 12:45 PM, Darius Buntinas wrote: >> >>> >>> On Mar 1, 2011, at 8:30 AM, Joshua Hursey wrote: >>> >>>> One side note that we should make explicit is that "L_i" can be >>>> updated in two ways. First, when an application calls >>>> MPI_comm_validate_local() to update "L_i" with all locally known >>>> failures. Secondly, if a failure is detected during any MPI operation >>>> (except the validate accessor functions) it will also update "L_i". So >>>> if we do a MPI_Send(rank=2), and rank 2 fails during the send then we >>>> want to make sure that if the user asks for the state of rank 2 it is >>>> identified as failed and not active. The application is implicitly >>>> noticing the update to "L_i" from the return code of the MPI_Send >>>> operation. >>>> >>>> Do folks see a problem with this additional semantic? >>> >>> This could present a problem with get_num_state and get_state in a >>> multithreaded environment. Of course if one has only one thread per >>> communicator, then it's OK, but is that realistic? The idea of L_i and >>> G being constant between validate calls was to avoid races like this. >>> >>> The user should understand that the way to get the state of a process >>> is >>> validate_local();get_state_rank() >>> or >>> validate_local();get_num_state();malloc();get_state() >>> And if the process has multiple threads using the same communicator, it >>> can synchronize access to validate_local as appropriate. >>> >>> I think the _local functions can be considered a convenience for the >>> user, since the user could keep track of L_i herself using the _global >>> values and noticing failed sends/receives. So if we look at it that >>> way, the fact that get_state doesn't report rank 2 (in your example >>> above) as having failed immediately after the send, might be OK. >> >> It just seems a bit odd as a semantic for the interfaces. But I >> understand your point. >> >> So the situation is (in shorthand): >> ------------------------ >> validate_local(comm); >> get_state_rank(comm, 2, state) /* state=OK; */ >> >> /*** 2 fails ***/ >> >> ret = MPI_Send(comm, 2); /* Error */ >> if( ERR_FAIL_STOP == ret ) { >> get_state_rank(comm, 2, state) /* state=OK; */ >> validate_local(comm); >> get_state_rank(comm, 2, state) /* state=FAILED; */ >> } >> ------------------------ >> >> instead of: >> ------------------------ >> validate_local(comm); >> get_state_rank(comm, 2, state) /* state=OK; */ >> >> /*** 2 fails ***/ >> >> ret = MPI_Send(comm, 2); /* Error */ >> if( ERR_FAIL_STOP == ret ) { >> get_state_rank(comm, 2, state) /* state=FAILED; */ >> } >> ------------------------ >> >> So the MPI implementation must keep another list of failed processes >> known to the MPI implementation (call it "H_i" for Hidden at Process i), >> but not yet made available in "L_i". So the MPI implementation checks >> "H_i" to determine if the MPI_Send() should fail. "H_i" represents the >> set of additional failures not in "G" or "L_i" at some time T for Process >> i. We can use list projections (similar to how "L_i" can be a physically >> smaller list than "G") for representing "H_i" to reduce the memory >> impact, but this does mean that there is slightly more work on the MPI >> internals side of things. >> >> So the programming practice that we are advocating is that before any >> get_state() operation that the user call validate_local() - or more >> precisely synchronize their local/global view of the state of the >> processes on the communicator. Right? >> >> -- Josh >> >> >>> >>> -d >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_Comm_validate parameters
by Darius Buntinas 03 Mar '11

03 Mar '11

Yeah. Maybe something emphasizing the difference between "getting" L_i (i.e., validate) and reading it (get_state). -d On Mar 3, 2011, at 11:20 AM, Joshua Hursey wrote: > So I think this is fine. Since it is a bit awkward, we should have a brief 'rationale' talking about why we went with this semantic.

1 0

Re: [Mpi3-ft] MPI_Comm_validate parameters
by Joshua Hursey 03 Mar '11

03 Mar '11

On Mar 2, 2011, at 12:11 PM, Darius Buntinas wrote: > > On Mar 2, 2011, at 8:51 AM, Joshua Hursey wrote: > >> So the MPI implementation must keep another list of failed processes known to the MPI implementation (call it "H_i" for Hidden at Process i), but not yet made available in "L_i". So the MPI implementation checks "H_i" to determine if the MPI_Send() should fail. "H_i" represents the set of additional failures not in "G" or "L_i" at some time T for Process i. We can use list projections (similar to how "L_i" can be a physically smaller list than "G") for representing "H_i" to reduce the memory impact, but this does mean that there is slightly more work on the MPI internals side of things. > > The way I would implement it is to represent H_i internally by setting the state of VCs in H_i to FAILED. Then have lists to represent G and L_i. For optimization so that we wouldn't have to scan the list of VCs when we do a comm_validate, I would probably also have a list representing L_i-H_i. > >> So the programming practice that we are advocating is that before any get_state() operation that the user call validate_local() - or more precisely synchronize their local/global view of the state of the processes on the communicator. Right? > > Right. I agree it seems awkward, but I think something is needed to avoid the get_num;malloc;get_state race with threads. Are there other solutions? The only other solution would be a mprobe like approach, but that has us introducing yet another object that the user has to manage. So I like that a bit less. So I think this is fine. Since it is a bit awkward, we should have a brief 'rationale' talking about why we went with this semantic. -- Josh > > -d > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Fault Tolerance (sub)Chapter or Tighter Integration
by Joshua Hursey 03 Mar '11

03 Mar '11

Yeah, that is along the lines I was thinking. It might be useful to do a plenary session to introduce the forum to the general concepts that we are playing with at the moment, and get some initial feedback. Maybe introduce the run-through stabilization proposal after we start to get a feel for the recovery proposal. But not necessary a plenary session as a means to a first reading. I think we need to have a good feel for the recovery semantics (implementation, application uses, ...) before we start thinking about preparing for a reading/vote. My intention with this email was just to get a feeling if folks have started thinking about what a standard PDF would look like with these additions. A clean PDF version of the developing proposal might be useful for someone new to see just the changes to the standard we are presenting, without all of our notes as with the wiki. I don't know if any of us have the cycles to start putting together the PDF, but if we did I was just curious if there was a preference on how to integrate the text. I kind of like the idea of adding a section to the Environmental Management chapter, then putting pointers to this chapter throughout the document as necessary. But I don't have a strong feeling one way or the other. Thanks, Josh On Mar 2, 2011, at 12:39 PM, Graham, Richard L. wrote: > Just to raise one point - I don't plan to bring this to the full forum > until we are further ahead on the implementation with respect to the > fault-recovery. This seemed to be the preference last time we brought the > fault-detection proposal to the full group. There is also merit on > getting further along with the fault recovery to see if there are any > items missed. > > Rich > > On 3/2/11 12:31 PM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: > >> >> I think a section in the environmental management chapter would make >> sense. Then we wouldn't need additional text in the Point to Point >> chapter for things like MPI_Send and MPI_Recv, but places where >> additional explanation is needed (perhaps collectives?) we would add in >> those chapters. >> >> Though I would be OK with making it a chapter too. We should then move >> Error Handling from the Environmental Management there too. (I think >> that's what you said) >> >> -d >> >> On Mar 1, 2011, at 8:38 AM, Joshua Hursey wrote: >> >>> We start edging toward a final draft of the run-through stabilization >>> proposal and embark on process recovery (TBA). As we do so, I wanted to >>> start thinking about how we might integrate this language into the >>> current MPI standard. A PDF version of the working proposal will make it >>> easier for someone new to pick up and read exactly what we are going to >>> add. This is in contrast to the mixture of notes and standard text that >>> is currently on the wiki. >>> >>> In particular, should we: >>> A) Create an entirely new chapter on Fault Tolerance and Error >>> Management. Pull in all existing section to a central location. >>> B) Add a section to the Environmental Management chapter on Fault >>> Tolerance. Pull in relevant existing sections on error handling into >>> this section. >>> C) Tightly integrate the semantics throughout the MPI standard (e.g., >>> P2P semantics in the P2P chapter, Collective semantics in the >>> Collectives chapter). >>> D) Something else... >>> >>> There are pros and cons to each. In essence the question is, should we >>> move all the error management logic to a central location or keep it >>> close to the actual functionality? >>> >>> What do folks think about this? >>> >>> -- Josh >>> >>> ------------------------------------ >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Fault Tolerance (sub)Chapter or Tighter Integration
by Graham, Richard L. 02 Mar '11

02 Mar '11

Just to raise one point - I don't plan to bring this to the full forum until we are further ahead on the implementation with respect to the fault-recovery. This seemed to be the preference last time we brought the fault-detection proposal to the full group. There is also merit on getting further along with the fault recovery to see if there are any items missed. Rich On 3/2/11 12:31 PM, "Darius Buntinas" <buntinas(a)mcs.anl.gov> wrote: > >I think a section in the environmental management chapter would make >sense. Then we wouldn't need additional text in the Point to Point >chapter for things like MPI_Send and MPI_Recv, but places where >additional explanation is needed (perhaps collectives?) we would add in >those chapters. > >Though I would be OK with making it a chapter too. We should then move >Error Handling from the Environmental Management there too. (I think >that's what you said) > >-d > >On Mar 1, 2011, at 8:38 AM, Joshua Hursey wrote: > >> We start edging toward a final draft of the run-through stabilization >>proposal and embark on process recovery (TBA). As we do so, I wanted to >>start thinking about how we might integrate this language into the >>current MPI standard. A PDF version of the working proposal will make it >>easier for someone new to pick up and read exactly what we are going to >>add. This is in contrast to the mixture of notes and standard text that >>is currently on the wiki. >> >> In particular, should we: >> A) Create an entirely new chapter on Fault Tolerance and Error >>Management. Pull in all existing section to a central location. >> B) Add a section to the Environmental Management chapter on Fault >>Tolerance. Pull in relevant existing sections on error handling into >>this section. >> C) Tightly integrate the semantics throughout the MPI standard (e.g., >>P2P semantics in the P2P chapter, Collective semantics in the >>Collectives chapter). >> D) Something else... >> >> There are pros and cons to each. In essence the question is, should we >>move all the error management logic to a central location or keep it >>close to the actual functionality? >> >> What do folks think about this? >> >> -- Josh >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Fault Tolerance (sub)Chapter or Tighter Integration
by Darius Buntinas 02 Mar '11

02 Mar '11

I think a section in the environmental management chapter would make sense. Then we wouldn't need additional text in the Point to Point chapter for things like MPI_Send and MPI_Recv, but places where additional explanation is needed (perhaps collectives?) we would add in those chapters. Though I would be OK with making it a chapter too. We should then move Error Handling from the Environmental Management there too. (I think that's what you said) -d On Mar 1, 2011, at 8:38 AM, Joshua Hursey wrote: > We start edging toward a final draft of the run-through stabilization proposal and embark on process recovery (TBA). As we do so, I wanted to start thinking about how we might integrate this language into the current MPI standard. A PDF version of the working proposal will make it easier for someone new to pick up and read exactly what we are going to add. This is in contrast to the mixture of notes and standard text that is currently on the wiki. > > In particular, should we: > A) Create an entirely new chapter on Fault Tolerance and Error Management. Pull in all existing section to a central location. > B) Add a section to the Environmental Management chapter on Fault Tolerance. Pull in relevant existing sections on error handling into this section. > C) Tightly integrate the semantics throughout the MPI standard (e.g., P2P semantics in the P2P chapter, Collective semantics in the Collectives chapter). > D) Something else... > > There are pros and cons to each. In essence the question is, should we move all the error management logic to a central location or keep it close to the actual functionality? > > What do folks think about this? > > -- Josh > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] MPI_Comm_validate parameters
by Darius Buntinas 02 Mar '11

02 Mar '11

On Mar 2, 2011, at 8:51 AM, Joshua Hursey wrote: > So the MPI implementation must keep another list of failed processes known to the MPI implementation (call it "H_i" for Hidden at Process i), but not yet made available in "L_i". So the MPI implementation checks "H_i" to determine if the MPI_Send() should fail. "H_i" represents the set of additional failures not in "G" or "L_i" at some time T for Process i. We can use list projections (similar to how "L_i" can be a physically smaller list than "G") for representing "H_i" to reduce the memory impact, but this does mean that there is slightly more work on the MPI internals side of things. The way I would implement it is to represent H_i internally by setting the state of VCs in H_i to FAILED. Then have lists to represent G and L_i. For optimization so that we wouldn't have to scan the list of VCs when we do a comm_validate, I would probably also have a list representing L_i-H_i. > So the programming practice that we are advocating is that before any get_state() operation that the user call validate_local() - or more precisely synchronize their local/global view of the state of the processes on the communicator. Right? Right. I agree it seems awkward, but I think something is needed to avoid the get_num;malloc;get_state race with threads. Are there other solutions? -d

1 0

Re: [Mpi3-ft] MPI_Comm_validate parameters
by Graham, Richard L. 02 Mar '11

02 Mar '11

I believe that what is being proposed here is not thread-safe. If you look back at some of the docs that were put together when we were actively looking at this over a year ago, there was a proposal on how to do this in a thread safe manner. Having said that, I am not sure if thread safety is an issue here or not - it depends very much on the life-cycle of the state information. In the original proposal, the "vector" returned by comm_validate had some state information. Rich On 3/2/11 9:51 AM, "Joshua Hursey" <jjhursey(a)open-mpi.org> wrote: > >On Mar 1, 2011, at 12:45 PM, Darius Buntinas wrote: > >> >> On Mar 1, 2011, at 8:30 AM, Joshua Hursey wrote: >> >>> One side note that we should make explicit is that "L_i" can be >>>updated in two ways. First, when an application calls >>>MPI_comm_validate_local() to update "L_i" with all locally known >>>failures. Secondly, if a failure is detected during any MPI operation >>>(except the validate accessor functions) it will also update "L_i". So >>>if we do a MPI_Send(rank=2), and rank 2 fails during the send then we >>>want to make sure that if the user asks for the state of rank 2 it is >>>identified as failed and not active. The application is implicitly >>>noticing the update to "L_i" from the return code of the MPI_Send >>>operation. >>> >>> Do folks see a problem with this additional semantic? >> >> This could present a problem with get_num_state and get_state in a >>multithreaded environment. Of course if one has only one thread per >>communicator, then it's OK, but is that realistic? The idea of L_i and >>G being constant between validate calls was to avoid races like this. >> >> The user should understand that the way to get the state of a process >>is >> validate_local();get_state_rank() >> or >> validate_local();get_num_state();malloc();get_state() >> And if the process has multiple threads using the same communicator, it >>can synchronize access to validate_local as appropriate. >> >> I think the _local functions can be considered a convenience for the >>user, since the user could keep track of L_i herself using the _global >>values and noticing failed sends/receives. So if we look at it that >>way, the fact that get_state doesn't report rank 2 (in your example >>above) as having failed immediately after the send, might be OK. > >It just seems a bit odd as a semantic for the interfaces. But I >understand your point. > >So the situation is (in shorthand): >------------------------ >validate_local(comm); >get_state_rank(comm, 2, state) /* state=OK; */ > >/*** 2 fails ***/ > >ret = MPI_Send(comm, 2); /* Error */ >if( ERR_FAIL_STOP == ret ) { > get_state_rank(comm, 2, state) /* state=OK; */ > validate_local(comm); > get_state_rank(comm, 2, state) /* state=FAILED; */ >} >------------------------ > >instead of: >------------------------ >validate_local(comm); >get_state_rank(comm, 2, state) /* state=OK; */ > >/*** 2 fails ***/ > >ret = MPI_Send(comm, 2); /* Error */ >if( ERR_FAIL_STOP == ret ) { > get_state_rank(comm, 2, state) /* state=FAILED; */ >} >------------------------ > >So the MPI implementation must keep another list of failed processes >known to the MPI implementation (call it "H_i" for Hidden at Process i), >but not yet made available in "L_i". So the MPI implementation checks >"H_i" to determine if the MPI_Send() should fail. "H_i" represents the >set of additional failures not in "G" or "L_i" at some time T for Process >i. We can use list projections (similar to how "L_i" can be a physically >smaller list than "G") for representing "H_i" to reduce the memory >impact, but this does mean that there is slightly more work on the MPI >internals side of things. > >So the programming practice that we are advocating is that before any >get_state() operation that the user call validate_local() - or more >precisely synchronize their local/global view of the state of the >processes on the communicator. Right? > >-- Josh > > >> >> -d >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > >------------------------------------ >Joshua Hursey >Postdoctoral Research Associate >Oak Ridge National Laboratory >http://users.nccs.gov/~jjhursey > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Fault Tolerance (sub)Chapter or Tighter Integration
by Toon Knapen 02 Mar '11

02 Mar '11

(I'm just listening in on all conversations concerning FT because I need this kind of functionality in my app but I'm not an expert, so I'll give you my opinion from an average-user-pov) I think it would be most clear and helpfull to have a seperate chapter on all issues concerning FT. Nevertheless references to this chapter should be integrated along the semantics throughout the whole standard. Thus a central approach but with pointers to the 'central info' distributed all over the standard. On Tue, Mar 1, 2011 at 3:38 PM, Joshua Hursey <jjhursey(a)open-mpi.org> wrote: > We start edging toward a final draft of the run-through stabilization > proposal and embark on process recovery (TBA). As we do so, I wanted to > start thinking about how we might integrate this language into the current > MPI standard. A PDF version of the working proposal will make it easier for > someone new to pick up and read exactly what we are going to add. This is in > contrast to the mixture of notes and standard text that is currently on the > wiki. > > In particular, should we: > A) Create an entirely new chapter on Fault Tolerance and Error Management. > Pull in all existing section to a central location. > B) Add a section to the Environmental Management chapter on Fault > Tolerance. Pull in relevant existing sections on error handling into this > section. > C) Tightly integrate the semantics throughout the MPI standard (e.g., P2P > semantics in the P2P chapter, Collective semantics in the Collectives > chapter). > D) Something else... > > There are pros and cons to each. In essence the question is, should we move > all the error management logic to a central location or keep it close to the > actual functionality? > > What do folks think about this? > > -- Josh > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] MPI_Comm_validate parameters
by Joshua Hursey 02 Mar '11

02 Mar '11

On Mar 1, 2011, at 12:45 PM, Darius Buntinas wrote: > > On Mar 1, 2011, at 8:30 AM, Joshua Hursey wrote: > >> One side note that we should make explicit is that "L_i" can be updated in two ways. First, when an application calls MPI_comm_validate_local() to update "L_i" with all locally known failures. Secondly, if a failure is detected during any MPI operation (except the validate accessor functions) it will also update "L_i". So if we do a MPI_Send(rank=2), and rank 2 fails during the send then we want to make sure that if the user asks for the state of rank 2 it is identified as failed and not active. The application is implicitly noticing the update to "L_i" from the return code of the MPI_Send operation. >> >> Do folks see a problem with this additional semantic? > > This could present a problem with get_num_state and get_state in a multithreaded environment. Of course if one has only one thread per communicator, then it's OK, but is that realistic? The idea of L_i and G being constant between validate calls was to avoid races like this. > > The user should understand that the way to get the state of a process is > validate_local();get_state_rank() > or > validate_local();get_num_state();malloc();get_state() > And if the process has multiple threads using the same communicator, it can synchronize access to validate_local as appropriate. > > I think the _local functions can be considered a convenience for the user, since the user could keep track of L_i herself using the _global values and noticing failed sends/receives. So if we look at it that way, the fact that get_state doesn't report rank 2 (in your example above) as having failed immediately after the send, might be OK. It just seems a bit odd as a semantic for the interfaces. But I understand your point. So the situation is (in shorthand): ------------------------ validate_local(comm); get_state_rank(comm, 2, state) /* state=OK; */ /*** 2 fails ***/ ret = MPI_Send(comm, 2); /* Error */ if( ERR_FAIL_STOP == ret ) { get_state_rank(comm, 2, state) /* state=OK; */ validate_local(comm); get_state_rank(comm, 2, state) /* state=FAILED; */ } ------------------------ instead of: ------------------------ validate_local(comm); get_state_rank(comm, 2, state) /* state=OK; */ /*** 2 fails ***/ ret = MPI_Send(comm, 2); /* Error */ if( ERR_FAIL_STOP == ret ) { get_state_rank(comm, 2, state) /* state=FAILED; */ } ------------------------ So the MPI implementation must keep another list of failed processes known to the MPI implementation (call it "H_i" for Hidden at Process i), but not yet made available in "L_i". So the MPI implementation checks "H_i" to determine if the MPI_Send() should fail. "H_i" represents the set of additional failures not in "G" or "L_i" at some time T for Process i. We can use list projections (similar to how "L_i" can be a physically smaller list than "G") for representing "H_i" to reduce the memory impact, but this does mean that there is slightly more work on the MPI internals side of things. So the programming practice that we are advocating is that before any get_state() operation that the user call validate_local() - or more precisely synchronize their local/global view of the state of the processes on the communicator. Right? -- Josh > > -d > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0