August 2010 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] MPI_Init / MPI_Finalize
by Joshua Hursey 26 Aug '10

26 Aug '10

Bronis, thanks for the clarification on MPI_Finalize. I guess the detail that I was trying to get at is that it is not specified whether MPI_Finalize is a leave early/enter late kind of collective. So, at least the way I read it, it would be valid if one process enters and exits MPI_Finalize before a different process enters MPI_Finalize (similar to MPI_Bcast). This points to the reasoning behind the advice to implementors just above the cited paragraph that suggests a barrier operation during MPI_Finalize as one option. Am I interpreting this correctly? One misleading sentence to me is the following on p291 just after the definition of MPI_Finalize: "Each process must call MPI_FINALIZE before it exits." The 'it' is slightly unclear. I think this is referring to the process exiting, not the function. If 'it' referred to the function then this would disallow the collective to have some ranks leave the collective before all have joined, so requiring a barrier semantic. For the fault tolerance discussion, I think the question is on the consistency of the return code. Should the return code have commit/abort properties? So if there is a success then all processes return success. If some process fails during MPI_Finalize, should all processes return some error code. It is unclear to me if this property (commit/abort return codes) is really useful to the application. If all processes return success then a process fails directly afterwards, the other processes have no way of being notified. So what action could the remaining processes realistically take in either the success or failure case. So my suggestion is that we allow MPI_Finalize to preserve its loose synchrony, leave early collective property as long as the rank is no longer needed to continue interacting with any of the connected process (say for relaying error information). This means that some ranks may return success while other return error if a process fails during finalize. MPI implementations may choose to provide applications with commit/abort semantics, but are not required to do so. Does that sounds reasonable for MPI_Finalize? Thanks, Josh On Aug 26, 2010, at 12:15 AM, Fab Tillier wrote: > Bronis R. de Supinski wrote on Wed, 25 Aug 2010 at 21:08:36 > >> >> Fab: >> >> There is no wiggle room. MPI_FINALIZE is collective across >> MPI_COMM_WORLD. I do not understand why you would say otherwise. >> Here is more of the passage I was quoting: >> >> ----------------- >> >> MPI_FINALIZE is collective over all connected processes. If no processes >> were spawned, accepted or connected then this means over MPI_COMM_WORLD; > > Ahh, I missed this part, sorry. > > -Fab > >> otherwise it is collective over the union of all processes that have >> been and continue to be connected, as explained in Section Releasing >> Connections on page Releasing Connections. >> >> ----------------- >> >> The "connected" terminology is used to handle dynamic process >> management issues, for which the set of all processes cannot >> easily be defined in terms of a single communicator. >> >> Bronis >> >> >> >> >> On Wed, 25 Aug 2010, Fab Tillier wrote: >> >>> What defines "connected"? MPI_FINALIZE isn't collective across >> MPI_COMM_WORLD, as processes might never communicate with one another. >> Even if they do, communication may not require a connection, so they >> may never be connected. >>> >>> It seems to me there might be enough wiggle room in the standard to >>> allow MPI_Finalize to not be collective at all? >>> >>> -Fab >>> >>> Bronis R. de Supinski wrote on Wed, 25 Aug 2010 at 15:06:38 >>> >>>> >>>> Josh: >>>> >>>> On p293 of the 2.2 standard, it says "MPI_FINALIZE is collective >>>> over all connected processes." I don't know that the call being >>>> collective changes your analysis but your statement that the >>>> call is not collective was incorrect... >>>> >>>> Bronis >>>> >>>> >>>> On Wed, 25 Aug 2010, Joshua Hursey wrote: >>>> >>>>> During the discussion of the run-though stabilization proposal today >>>>> on the teleconf, we spent a while discussing the expected behavior of >>>>> MPI_Init and MPI_Finalize in the presence of process failures. I >>>>> would like to broaden the discussion a bit to help pin down the >>>>> expected behavior. >>>>> >>>>> MPI_Init(): ----------- Problem: If a process fails before or during >>>>> MPI_Init, what should the MPI implementation do? >>>>> >>>>> The current standard says nothing about the return value of >>>>> MPI_Init() (Ch. 8.7). To the greatest possible extent the application >>>>> should not be put in danger if it wishes to ignore errors (assumes >>>>> MPI_ERRORS_ARE_FATAL), so returning an error from this function (in >>>>> contrast to aborting the job) might be dangerous. However, if the >>>>> application is prepared to handle process failures, it is unable to >>>>> communicate that information to the MPI implementation until after >>>>> the completion of MPI_Init(). >>>>> >>>>> So a couple of solutions were presented each with pros and cons >>>>> (please fill in if I missed any): 1) If a process fails in MPI_Init() >>>>> (default error handler is MPI_ERRORS_ARE_FATAL) then the entire job >>>>> is aborted (similar to calling MPI_Abort on MPI_COMM_WORLD). >>>>> >>>>> 2) If a process fails in MPI_Init() the MPI implementation will >>>>> return an appropriate error code/class (e.g., >>>>> MPI_ERR_RANK_FAIL_STOP), and all subsequent calls into the MPI >>>>> implementation will return the error class MPI_ERR_OTHER (should be >>>>> create a MPI_ERR_NOT_ACTIVE?). Applications should eventually notice >>>>> the error and terminate. >>>>> >>>>> 3) Allow the application to register only the MPI_ERRORS_RETURN >>>>> handle on MPI_COMM_WORLD before MPI_Init() using the >>>>> MPI_Errhandler_set() function. Errors that occur before the >>>>> MPI_Errhandler_set() call are fatal. Errors afterward, including >>>>> during MPI_Init() are not fatal. >>>>> >>>>> In the cases where MPI_Init() returns MPI_ERR_RANK_FAIL_STOP to >>>>> indicate a process failure, is the library usable or not? If the >>>>> application can continue running through the failure, then the MPI >>>>> library should still be usable, thus MPI_Init() must be fault >>>>> tolerant in its initialization to be able to handle process failures. >>>>> If the MPI implementation finds itself in trouble and cannot continue >>>>> it should return MPI_ERR_CANNOT_CONTINUE from all subsequent calls >>>>> including MPI_Init, if possible. >>>>> >>>>> >>>>> MPI_Finalize(): --------------- Problem: If a process fails before or >>>>> during MPI_Finalize (and the error handler is not >>>>> MPI_ERRORS_ARE_FATAL), what should this function return? Should that >>>>> return value be consistent to all processes? >>>>> >>>>> To preserve locality of fault handling, a local process should not be >>>>> explicitly forced to recognize the failure of a peer process that >>>>> they never interact with neither directly (e.g., point-to-point) or >>>>> indirectly (e.g., collective). So MPI_Finalize should be fault >>>>> tolerant and keep trying to complete even in the presence of failures. >>>>> >>>>> MPI_Finalize is not required to be a collective operation, though it >>>>> is often implemented that way. An implementation may need to delay >>>>> the return from MPI_Finalize until its role in the failure >>>>> information distribution channel is complete. But we should not >>>>> require a multi- phase commit protocol to ensure that everyone either >>>>> succeeds or returns some error. Implementations may do so internally >>>>> in order to ensure that MPI_Finalize does not hang. >>>>> >>>>> If MPI_Finalize returns an error (say MPI_ERR_RANK_FAIL_STOP >>>>> indicating a 'new to this rank' failure), what good is this >>>>> information to the application? It cannot query for which rank(s) >>>>> failed since MPI has been finalized. Nor can it initiate recovery. >>>>> The best it could do is assume that all other processes failed and >>>>> take local action. >>>>> >>>>> >>>>> MPI_Finalize: MPI_COMM_WORLD process rank 0: >>>>> -------------------------------------------- In chapter 8, Example >>>>> 8.7 illustrates that "Although it is not required that all processes >>>>> return from MPI_Finalize, it is required that at least process 0 in >>>>> MPI_COMM_WORLD return, so that users can know that the MPI portion of >>>>> the computation is over." >>>>> >>>>> We deduced that the reasoning for this explanation was to allow for >>>>> MPI implementation that create and destroy MPI processes during >>>>> init/finalize from rank 0. Or worded differently, rank 0 is the only >>>>> rank that can be assumed to exist before MPI_Init and after >>>>> MPI_Finalize. >>>>> >>>>> Problem: So what if rank 0 fails at some point during the computation >>>>> (or just some point during MPI_Finalize)? >>>>> >>>>> In the proposal, I added an advice to users to tell them to not >>>>> depend on any specific ranks to exist before MPI_Init or after >>>>> MPI_Finalize. So, in a faulty environment, the example will produce >>>>> incorrect results under certain failure scenarios (e.g., failure of >>>>> rank 0). >>>>> >>>>> In an MPI environment that depends on rank 0 for process creation and >>>>> destruction, the failure of rank 0 is (should be?) critical and the >>>>> MPI implementation will either abort the job or return >>>>> MPI_ERR_CANNOT_CONTINUE from all calls to the MPI implementation. So >>>>> we believe that the advice to users was a sufficient addition to this >>>>> section. What do others think? >>>>> >>>>> >>>>> So MPI_Init seems to be a more complex issue than MPI_Finalize. What >>>>> do folks think about the presented problems and possible solutions? >>>>> Are there other issues not mentioned here that we should be >>>>> addressing? >>>>> >>>>> -- Josh >>>>> >>>>> Run-Through Stabilization Proposal: >>>>> https://**svn.mpi-forum.org/trac/mpi-forum- >>>>> web/wiki/ft/run_through_stabilization >>>>> >>>>> ------------------------------------ >>>>> Joshua Hursey >>>>> Postdoctoral Research Associate >>>>> Oak Ridge National Laboratory >>>>> http://**www.**cs.indiana.edu/~jjhursey >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] MPI_Init / MPI_Finalize
by Fab Tillier 25 Aug '10

25 Aug '10

Bronis R. de Supinski wrote on Wed, 25 Aug 2010 at 21:08:36 > > Fab: > > There is no wiggle room. MPI_FINALIZE is collective across > MPI_COMM_WORLD. I do not understand why you would say otherwise. > Here is more of the passage I was quoting: > > ----------------- > > MPI_FINALIZE is collective over all connected processes. If no processes > were spawned, accepted or connected then this means over MPI_COMM_WORLD; Ahh, I missed this part, sorry. -Fab > otherwise it is collective over the union of all processes that have > been and continue to be connected, as explained in Section Releasing > Connections on page Releasing Connections. > > ----------------- > > The "connected" terminology is used to handle dynamic process > management issues, for which the set of all processes cannot > easily be defined in terms of a single communicator. > > Bronis > > > > > On Wed, 25 Aug 2010, Fab Tillier wrote: > >> What defines "connected"? MPI_FINALIZE isn't collective across > MPI_COMM_WORLD, as processes might never communicate with one another. > Even if they do, communication may not require a connection, so they > may never be connected. >> >> It seems to me there might be enough wiggle room in the standard to >> allow MPI_Finalize to not be collective at all? >> >> -Fab >> >> Bronis R. de Supinski wrote on Wed, 25 Aug 2010 at 15:06:38 >> >>> >>> Josh: >>> >>> On p293 of the 2.2 standard, it says "MPI_FINALIZE is collective >>> over all connected processes." I don't know that the call being >>> collective changes your analysis but your statement that the >>> call is not collective was incorrect... >>> >>> Bronis >>> >>> >>> On Wed, 25 Aug 2010, Joshua Hursey wrote: >>> >>>> During the discussion of the run-though stabilization proposal today >>>> on the teleconf, we spent a while discussing the expected behavior of >>>> MPI_Init and MPI_Finalize in the presence of process failures. I >>>> would like to broaden the discussion a bit to help pin down the >>>> expected behavior. >>>> >>>> MPI_Init(): ----------- Problem: If a process fails before or during >>>> MPI_Init, what should the MPI implementation do? >>>> >>>> The current standard says nothing about the return value of >>>> MPI_Init() (Ch. 8.7). To the greatest possible extent the application >>>> should not be put in danger if it wishes to ignore errors (assumes >>>> MPI_ERRORS_ARE_FATAL), so returning an error from this function (in >>>> contrast to aborting the job) might be dangerous. However, if the >>>> application is prepared to handle process failures, it is unable to >>>> communicate that information to the MPI implementation until after >>>> the completion of MPI_Init(). >>>> >>>> So a couple of solutions were presented each with pros and cons >>>> (please fill in if I missed any): 1) If a process fails in MPI_Init() >>>> (default error handler is MPI_ERRORS_ARE_FATAL) then the entire job >>>> is aborted (similar to calling MPI_Abort on MPI_COMM_WORLD). >>>> >>>> 2) If a process fails in MPI_Init() the MPI implementation will >>>> return an appropriate error code/class (e.g., >>>> MPI_ERR_RANK_FAIL_STOP), and all subsequent calls into the MPI >>>> implementation will return the error class MPI_ERR_OTHER (should be >>>> create a MPI_ERR_NOT_ACTIVE?). Applications should eventually notice >>>> the error and terminate. >>>> >>>> 3) Allow the application to register only the MPI_ERRORS_RETURN >>>> handle on MPI_COMM_WORLD before MPI_Init() using the >>>> MPI_Errhandler_set() function. Errors that occur before the >>>> MPI_Errhandler_set() call are fatal. Errors afterward, including >>>> during MPI_Init() are not fatal. >>>> >>>> In the cases where MPI_Init() returns MPI_ERR_RANK_FAIL_STOP to >>>> indicate a process failure, is the library usable or not? If the >>>> application can continue running through the failure, then the MPI >>>> library should still be usable, thus MPI_Init() must be fault >>>> tolerant in its initialization to be able to handle process failures. >>>> If the MPI implementation finds itself in trouble and cannot continue >>>> it should return MPI_ERR_CANNOT_CONTINUE from all subsequent calls >>>> including MPI_Init, if possible. >>>> >>>> >>>> MPI_Finalize(): --------------- Problem: If a process fails before or >>>> during MPI_Finalize (and the error handler is not >>>> MPI_ERRORS_ARE_FATAL), what should this function return? Should that >>>> return value be consistent to all processes? >>>> >>>> To preserve locality of fault handling, a local process should not be >>>> explicitly forced to recognize the failure of a peer process that >>>> they never interact with neither directly (e.g., point-to-point) or >>>> indirectly (e.g., collective). So MPI_Finalize should be fault >>>> tolerant and keep trying to complete even in the presence of failures. >>>> >>>> MPI_Finalize is not required to be a collective operation, though it >>>> is often implemented that way. An implementation may need to delay >>>> the return from MPI_Finalize until its role in the failure >>>> information distribution channel is complete. But we should not >>>> require a multi- phase commit protocol to ensure that everyone either >>>> succeeds or returns some error. Implementations may do so internally >>>> in order to ensure that MPI_Finalize does not hang. >>>> >>>> If MPI_Finalize returns an error (say MPI_ERR_RANK_FAIL_STOP >>>> indicating a 'new to this rank' failure), what good is this >>>> information to the application? It cannot query for which rank(s) >>>> failed since MPI has been finalized. Nor can it initiate recovery. >>>> The best it could do is assume that all other processes failed and >>>> take local action. >>>> >>>> >>>> MPI_Finalize: MPI_COMM_WORLD process rank 0: >>>> -------------------------------------------- In chapter 8, Example >>>> 8.7 illustrates that "Although it is not required that all processes >>>> return from MPI_Finalize, it is required that at least process 0 in >>>> MPI_COMM_WORLD return, so that users can know that the MPI portion of >>>> the computation is over." >>>> >>>> We deduced that the reasoning for this explanation was to allow for >>>> MPI implementation that create and destroy MPI processes during >>>> init/finalize from rank 0. Or worded differently, rank 0 is the only >>>> rank that can be assumed to exist before MPI_Init and after >>>> MPI_Finalize. >>>> >>>> Problem: So what if rank 0 fails at some point during the computation >>>> (or just some point during MPI_Finalize)? >>>> >>>> In the proposal, I added an advice to users to tell them to not >>>> depend on any specific ranks to exist before MPI_Init or after >>>> MPI_Finalize. So, in a faulty environment, the example will produce >>>> incorrect results under certain failure scenarios (e.g., failure of >>>> rank 0). >>>> >>>> In an MPI environment that depends on rank 0 for process creation and >>>> destruction, the failure of rank 0 is (should be?) critical and the >>>> MPI implementation will either abort the job or return >>>> MPI_ERR_CANNOT_CONTINUE from all calls to the MPI implementation. So >>>> we believe that the advice to users was a sufficient addition to this >>>> section. What do others think? >>>> >>>> >>>> So MPI_Init seems to be a more complex issue than MPI_Finalize. What >>>> do folks think about the presented problems and possible solutions? >>>> Are there other issues not mentioned here that we should be >>>> addressing? >>>> >>>> -- Josh >>>> >>>> Run-Through Stabilization Proposal: >>>> https://**svn.mpi-forum.org/trac/mpi-forum- >>>> web/wiki/ft/run_through_stabilization >>>> >>>> ------------------------------------ >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://**www.**cs.indiana.edu/~jjhursey >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >>

1 0

Re: [Mpi3-ft] MPI_Init / MPI_Finalize
by Fab Tillier 25 Aug '10

25 Aug '10

I think perhaps a new MPI_Init function is in order, one that takes as input an error handler for MPI_COMM_WORLD. I'd go a step further and have the function take as input parameter the major and minor version of MPI that the application is requesting. Something like: MPI_Init_version( int version, int* argc, char ***argv, MPI_Errhandler errhandler, int required, int* provided ); The version parameter would encode the major and minor version of the standard that the application expects. This allows semantic changes to existing APIs (i.e. preserving the function signature but changing the behavior) without requiring introducing new APIs which should help control the API footprint of the standard. For example, with this we could change the behavior of existing APIs to support run-through stabilization without violating previous versions of the standard. The errhandler would have to be one of the predefined error handlers and avoids the need to allow MPI_Comm_set_errhandler to be called (on MPI_COMM_WORLD only) before the MPI implementation is initialized. Thoughts? -Fab Joshua Hursey wrote on Wed, 25 Aug 2010 at 13:24:32 > During the discussion of the run-though stabilization proposal today on > the teleconf, we spent a while discussing the expected behavior of > MPI_Init and MPI_Finalize in the presence of process failures. I would > like to broaden the discussion a bit to help pin down the expected > behavior. > > MPI_Init(): > ----------- > Problem: If a process fails before or during MPI_Init, what should the > MPI implementation do? > > The current standard says nothing about the return value of MPI_Init() > (Ch. 8.7). To the greatest possible extent the application should not > be put in danger if it wishes to ignore errors (assumes > MPI_ERRORS_ARE_FATAL), so returning an error from this function (in > contrast to aborting the job) might be dangerous. However, if the > application is prepared to handle process failures, it is unable to > communicate that information to the MPI implementation until after the > completion of MPI_Init(). > > So a couple of solutions were presented each with pros and cons (please > fill in if I missed any): > 1) If a process fails in MPI_Init() (default error handler is > MPI_ERRORS_ARE_FATAL) then the entire job is aborted (similar to > calling MPI_Abort on MPI_COMM_WORLD). > > 2) If a process fails in MPI_Init() the MPI implementation will return > an appropriate error code/class (e.g., MPI_ERR_RANK_FAIL_STOP), and all > subsequent calls into the MPI implementation will return the error > class MPI_ERR_OTHER (should be create a MPI_ERR_NOT_ACTIVE?). > Applications should eventually notice the error and terminate. > > 3) Allow the application to register only the MPI_ERRORS_RETURN handle > on MPI_COMM_WORLD before MPI_Init() using the MPI_Errhandler_set() > function. Errors that occur before the MPI_Errhandler_set() call are > fatal. Errors afterward, including during MPI_Init() are not fatal. > > In the cases where MPI_Init() returns MPI_ERR_RANK_FAIL_STOP to > indicate a process failure, is the library usable or not? If the > application can continue running through the failure, then the MPI > library should still be usable, thus MPI_Init() must be fault tolerant > in its initialization to be able to handle process failures. If the MPI > implementation finds itself in trouble and cannot continue it should > return MPI_ERR_CANNOT_CONTINUE from all subsequent calls including > MPI_Init, if possible. > > > MPI_Finalize(): > --------------- > Problem: If a process fails before or during MPI_Finalize (and the > error handler is not MPI_ERRORS_ARE_FATAL), what should this function > return? Should that return value be consistent to all processes? > > To preserve locality of fault handling, a local process should not be > explicitly forced to recognize the failure of a peer process that they > never interact with neither directly (e.g., point-to-point) or > indirectly (e.g., collective). So MPI_Finalize should be fault tolerant > and keep trying to complete even in the presence of failures. > > MPI_Finalize is not required to be a collective operation, though it is > often implemented that way. An implementation may need to delay the > return from MPI_Finalize until its role in the failure information > distribution channel is complete. But we should not require a multi- > phase commit protocol to ensure that everyone either succeeds or > returns some error. Implementations may do so internally in order to > ensure that MPI_Finalize does not hang. > > If MPI_Finalize returns an error (say MPI_ERR_RANK_FAIL_STOP indicating > a 'new to this rank' failure), what good is this information to the > application? It cannot query for which rank(s) failed since MPI has > been finalized. Nor can it initiate recovery. The best it could do is > assume that all other processes failed and take local action. > > > MPI_Finalize: MPI_COMM_WORLD process rank 0: > -------------------------------------------- In chapter 8, Example 8.7 > illustrates that "Although it is not required that all processes return > from MPI_Finalize, it is required that at least process 0 in > MPI_COMM_WORLD return, so that users can know that the MPI portion of > the computation is over." > > We deduced that the reasoning for this explanation was to allow for MPI > implementation that create and destroy MPI processes during > init/finalize from rank 0. Or worded differently, rank 0 is the only > rank that can be assumed to exist before MPI_Init and after MPI_Finalize. > > Problem: So what if rank 0 fails at some point during the computation > (or just some point during MPI_Finalize)? > > In the proposal, I added an advice to users to tell them to not depend > on any specific ranks to exist before MPI_Init or after MPI_Finalize. > So, in a faulty environment, the example will produce incorrect results > under certain failure scenarios (e.g., failure of rank 0). > > In an MPI environment that depends on rank 0 for process creation and > destruction, the failure of rank 0 is (should be?) critical and the MPI > implementation will either abort the job or return > MPI_ERR_CANNOT_CONTINUE from all calls to the MPI implementation. So we > believe that the advice to users was a sufficient addition to this > section. What do others think? > > > So MPI_Init seems to be a more complex issue than MPI_Finalize. What do > folks think about the presented problems and possible solutions? Are > there other issues not mentioned here that we should be addressing? > > -- Josh > > Run-Through Stabilization Proposal: > https://svn.mpi-forum.org/trac/mpi-forum- > web/wiki/ft/run_through_stabilization > > ------------------------------------ Joshua Hursey Postdoctoral Research > Associate Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] MPI_Init / MPI_Finalize
by Bronis R. de Supinski 25 Aug '10

25 Aug '10

Fab: There is no wiggle room. MPI_FINALIZE is collective across MPI_COMM_WORLD. I do not understand why you would say otherwise. Here is more of the passage I was quoting: ----------------- MPI_FINALIZE is collective over all connected processes. If no processes were spawned, accepted or connected then this means over MPI_COMM_WORLD; otherwise it is collective over the union of all processes that have been and continue to be connected, as explained in Section Releasing Connections on page Releasing Connections. ----------------- The "connected" terminology is used to handle dynamic process management issues, for which the set of all processes cannot easily be defined in terms of a single communicator. Bronis On Wed, 25 Aug 2010, Fab Tillier wrote: > What defines "connected"? MPI_FINALIZE isn't collective across MPI_COMM_WORLD, as processes might never communicate with one another. Even if they do, communication may not require a connection, so they may never be connected. > > It seems to me there might be enough wiggle room in the standard to allow MPI_Finalize to not be collective at all? > > -Fab > > Bronis R. de Supinski wrote on Wed, 25 Aug 2010 at 15:06:38 > >> >> Josh: >> >> On p293 of the 2.2 standard, it says "MPI_FINALIZE is collective >> over all connected processes." I don't know that the call being >> collective changes your analysis but your statement that the >> call is not collective was incorrect... >> >> Bronis >> >> >> On Wed, 25 Aug 2010, Joshua Hursey wrote: >> >>> During the discussion of the run-though stabilization proposal today >>> on the teleconf, we spent a while discussing the expected behavior of >>> MPI_Init and MPI_Finalize in the presence of process failures. I would >>> like to broaden the discussion a bit to help pin down the expected >>> behavior. >>> >>> MPI_Init(): ----------- Problem: If a process fails before or during >>> MPI_Init, what should the MPI implementation do? >>> >>> The current standard says nothing about the return value of >>> MPI_Init() (Ch. 8.7). To the greatest possible extent the application >>> should not be put in danger if it wishes to ignore errors (assumes >>> MPI_ERRORS_ARE_FATAL), so returning an error from this function (in >>> contrast to aborting the job) might be dangerous. However, if the >>> application is prepared to handle process failures, it is unable to >>> communicate that information to the MPI implementation until after the >>> completion of MPI_Init(). >>> >>> So a couple of solutions were presented each with pros and cons (please >>> fill in if I missed any): 1) If a process fails in MPI_Init() (default >>> error handler is >>> MPI_ERRORS_ARE_FATAL) then the entire job is aborted (similar to >>> calling MPI_Abort on MPI_COMM_WORLD). >>> >>> 2) If a process fails in MPI_Init() the MPI implementation will >>> return an appropriate error code/class (e.g., MPI_ERR_RANK_FAIL_STOP), >>> and all subsequent calls into the MPI implementation will return the >>> error class MPI_ERR_OTHER (should be create a MPI_ERR_NOT_ACTIVE?). >>> Applications should eventually notice the error and terminate. >>> >>> 3) Allow the application to register only the MPI_ERRORS_RETURN >>> handle on MPI_COMM_WORLD before MPI_Init() using the >>> MPI_Errhandler_set() function. Errors that occur before the >>> MPI_Errhandler_set() call are fatal. Errors afterward, including during >>> MPI_Init() are not fatal. >>> >>> In the cases where MPI_Init() returns MPI_ERR_RANK_FAIL_STOP to >>> indicate a process failure, is the library usable or not? If the >>> application can continue running through the failure, then the MPI >>> library should still be usable, thus MPI_Init() must be fault tolerant >>> in its initialization to be able to handle process failures. If the MPI >>> implementation finds itself in trouble and cannot continue it should >>> return MPI_ERR_CANNOT_CONTINUE from all subsequent calls including >>> MPI_Init, if possible. >>> >>> >>> MPI_Finalize(): >>> --------------- >>> Problem: If a process fails before or during MPI_Finalize (and the >>> error handler is not MPI_ERRORS_ARE_FATAL), what should this function >>> return? Should that return value be consistent to all processes? >>> >>> To preserve locality of fault handling, a local process should not be >>> explicitly forced to recognize the failure of a peer process that they >>> never interact with neither directly (e.g., point-to-point) or >>> indirectly (e.g., collective). So MPI_Finalize should be fault tolerant >>> and keep trying to complete even in the presence of failures. >>> >>> MPI_Finalize is not required to be a collective operation, though it >>> is often implemented that way. An implementation may need to delay the >>> return from MPI_Finalize until its role in the failure information >>> distribution channel is complete. But we should not require a multi- >>> phase commit protocol to ensure that everyone either succeeds or >>> returns some error. Implementations may do so internally in order to >>> ensure that MPI_Finalize does not hang. >>> >>> If MPI_Finalize returns an error (say MPI_ERR_RANK_FAIL_STOP >>> indicating a 'new to this rank' failure), what good is this information >>> to the application? It cannot query for which rank(s) failed since MPI >>> has been finalized. Nor can it initiate recovery. The best it could do >>> is assume that all other processes failed and take local action. >>> >>> >>> MPI_Finalize: MPI_COMM_WORLD process rank 0: >>> -------------------------------------------- >>> In chapter 8, Example 8.7 illustrates that "Although it is not >>> required that all processes return from MPI_Finalize, it is required >>> that at least process 0 in MPI_COMM_WORLD return, so that users can >>> know that the MPI portion of the computation is over." >>> >>> We deduced that the reasoning for this explanation was to allow for >>> MPI implementation that create and destroy MPI processes during >>> init/finalize from rank 0. Or worded differently, rank 0 is the only >>> rank that can be assumed to exist before MPI_Init and after >>> MPI_Finalize. >>> >>> Problem: So what if rank 0 fails at some point during the computation >>> (or just some point during MPI_Finalize)? >>> >>> In the proposal, I added an advice to users to tell them to not >>> depend on any specific ranks to exist before MPI_Init or after >>> MPI_Finalize. So, in a faulty environment, the example will produce >>> incorrect results under certain failure scenarios (e.g., failure of >>> rank 0). >>> >>> In an MPI environment that depends on rank 0 for process creation and >>> destruction, the failure of rank 0 is (should be?) critical and the MPI >>> implementation will either abort the job or return >>> MPI_ERR_CANNOT_CONTINUE from all calls to the MPI implementation. So we >>> believe that the advice to users was a sufficient addition to this >>> section. What do others think? >>> >>> >>> So MPI_Init seems to be a more complex issue than MPI_Finalize. What >>> do folks think about the presented problems and possible solutions? Are >>> there other issues not mentioned here that we should be addressing? >>> >>> -- Josh >>> >>> Run-Through Stabilization Proposal: >>> https://**svn.mpi-forum.org/trac/mpi-forum- >>> web/wiki/ft/run_through_stabilization >>> >>> ------------------------------------ >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://**www.**cs.indiana.edu/~jjhursey >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] MPI_Init / MPI_Finalize
by Fab Tillier 25 Aug '10

25 Aug '10

What defines "connected"? MPI_FINALIZE isn't collective across MPI_COMM_WORLD, as processes might never communicate with one another. Even if they do, communication may not require a connection, so they may never be connected. It seems to me there might be enough wiggle room in the standard to allow MPI_Finalize to not be collective at all? -Fab Bronis R. de Supinski wrote on Wed, 25 Aug 2010 at 15:06:38 > > Josh: > > On p293 of the 2.2 standard, it says "MPI_FINALIZE is collective > over all connected processes." I don't know that the call being > collective changes your analysis but your statement that the > call is not collective was incorrect... > > Bronis > > > On Wed, 25 Aug 2010, Joshua Hursey wrote: > >> During the discussion of the run-though stabilization proposal today >> on the teleconf, we spent a while discussing the expected behavior of >> MPI_Init and MPI_Finalize in the presence of process failures. I would >> like to broaden the discussion a bit to help pin down the expected >> behavior. >> >> MPI_Init(): ----------- Problem: If a process fails before or during >> MPI_Init, what should the MPI implementation do? >> >> The current standard says nothing about the return value of >> MPI_Init() (Ch. 8.7). To the greatest possible extent the application >> should not be put in danger if it wishes to ignore errors (assumes >> MPI_ERRORS_ARE_FATAL), so returning an error from this function (in >> contrast to aborting the job) might be dangerous. However, if the >> application is prepared to handle process failures, it is unable to >> communicate that information to the MPI implementation until after the >> completion of MPI_Init(). >> >> So a couple of solutions were presented each with pros and cons (please >> fill in if I missed any): 1) If a process fails in MPI_Init() (default >> error handler is >> MPI_ERRORS_ARE_FATAL) then the entire job is aborted (similar to >> calling MPI_Abort on MPI_COMM_WORLD). >> >> 2) If a process fails in MPI_Init() the MPI implementation will >> return an appropriate error code/class (e.g., MPI_ERR_RANK_FAIL_STOP), >> and all subsequent calls into the MPI implementation will return the >> error class MPI_ERR_OTHER (should be create a MPI_ERR_NOT_ACTIVE?). >> Applications should eventually notice the error and terminate. >> >> 3) Allow the application to register only the MPI_ERRORS_RETURN >> handle on MPI_COMM_WORLD before MPI_Init() using the >> MPI_Errhandler_set() function. Errors that occur before the >> MPI_Errhandler_set() call are fatal. Errors afterward, including during >> MPI_Init() are not fatal. >> >> In the cases where MPI_Init() returns MPI_ERR_RANK_FAIL_STOP to >> indicate a process failure, is the library usable or not? If the >> application can continue running through the failure, then the MPI >> library should still be usable, thus MPI_Init() must be fault tolerant >> in its initialization to be able to handle process failures. If the MPI >> implementation finds itself in trouble and cannot continue it should >> return MPI_ERR_CANNOT_CONTINUE from all subsequent calls including >> MPI_Init, if possible. >> >> >> MPI_Finalize(): >> --------------- >> Problem: If a process fails before or during MPI_Finalize (and the >> error handler is not MPI_ERRORS_ARE_FATAL), what should this function >> return? Should that return value be consistent to all processes? >> >> To preserve locality of fault handling, a local process should not be >> explicitly forced to recognize the failure of a peer process that they >> never interact with neither directly (e.g., point-to-point) or >> indirectly (e.g., collective). So MPI_Finalize should be fault tolerant >> and keep trying to complete even in the presence of failures. >> >> MPI_Finalize is not required to be a collective operation, though it >> is often implemented that way. An implementation may need to delay the >> return from MPI_Finalize until its role in the failure information >> distribution channel is complete. But we should not require a multi- >> phase commit protocol to ensure that everyone either succeeds or >> returns some error. Implementations may do so internally in order to >> ensure that MPI_Finalize does not hang. >> >> If MPI_Finalize returns an error (say MPI_ERR_RANK_FAIL_STOP >> indicating a 'new to this rank' failure), what good is this information >> to the application? It cannot query for which rank(s) failed since MPI >> has been finalized. Nor can it initiate recovery. The best it could do >> is assume that all other processes failed and take local action. >> >> >> MPI_Finalize: MPI_COMM_WORLD process rank 0: >> -------------------------------------------- >> In chapter 8, Example 8.7 illustrates that "Although it is not >> required that all processes return from MPI_Finalize, it is required >> that at least process 0 in MPI_COMM_WORLD return, so that users can >> know that the MPI portion of the computation is over." >> >> We deduced that the reasoning for this explanation was to allow for >> MPI implementation that create and destroy MPI processes during >> init/finalize from rank 0. Or worded differently, rank 0 is the only >> rank that can be assumed to exist before MPI_Init and after >> MPI_Finalize. >> >> Problem: So what if rank 0 fails at some point during the computation >> (or just some point during MPI_Finalize)? >> >> In the proposal, I added an advice to users to tell them to not >> depend on any specific ranks to exist before MPI_Init or after >> MPI_Finalize. So, in a faulty environment, the example will produce >> incorrect results under certain failure scenarios (e.g., failure of >> rank 0). >> >> In an MPI environment that depends on rank 0 for process creation and >> destruction, the failure of rank 0 is (should be?) critical and the MPI >> implementation will either abort the job or return >> MPI_ERR_CANNOT_CONTINUE from all calls to the MPI implementation. So we >> believe that the advice to users was a sufficient addition to this >> section. What do others think? >> >> >> So MPI_Init seems to be a more complex issue than MPI_Finalize. What >> do folks think about the presented problems and possible solutions? Are >> there other issues not mentioned here that we should be addressing? >> >> -- Josh >> >> Run-Through Stabilization Proposal: >> https://*svn.mpi-forum.org/trac/mpi-forum- >> web/wiki/ft/run_through_stabilization >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://*www.*cs.indiana.edu/~jjhursey >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] MPI_Init / MPI_Finalize
by Bronis R. de Supinski 25 Aug '10

25 Aug '10

Josh: On p293 of the 2.2 standard, it says "MPI_FINALIZE is collective over all connected processes." I don't know that the call being collective changes your analysis but your statement that the call is not collective was incorrect... Bronis On Wed, 25 Aug 2010, Joshua Hursey wrote: > During the discussion of the run-though stabilization proposal today on the teleconf, we spent a while discussing the expected behavior of MPI_Init and MPI_Finalize in the presence of process failures. I would like to broaden the discussion a bit to help pin down the expected behavior. > > MPI_Init(): > ----------- > Problem: If a process fails before or during MPI_Init, what should the MPI implementation do? > > The current standard says nothing about the return value of MPI_Init() (Ch. 8.7). To the greatest possible extent the application should not be put in danger if it wishes to ignore errors (assumes MPI_ERRORS_ARE_FATAL), so returning an error from this function (in contrast to aborting the job) might be dangerous. However, if the application is prepared to handle process failures, it is unable to communicate that information to the MPI implementation until after the completion of MPI_Init(). > > So a couple of solutions were presented each with pros and cons (please fill in if I missed any): > 1) If a process fails in MPI_Init() (default error handler is MPI_ERRORS_ARE_FATAL) then the entire job is aborted (similar to calling MPI_Abort on MPI_COMM_WORLD). > > 2) If a process fails in MPI_Init() the MPI implementation will return an appropriate error code/class (e.g., MPI_ERR_RANK_FAIL_STOP), and all subsequent calls into the MPI implementation will return the error class MPI_ERR_OTHER (should be create a MPI_ERR_NOT_ACTIVE?). Applications should eventually notice the error and terminate. > > 3) Allow the application to register only the MPI_ERRORS_RETURN handle on MPI_COMM_WORLD before MPI_Init() using the MPI_Errhandler_set() function. Errors that occur before the MPI_Errhandler_set() call are fatal. Errors afterward, including during MPI_Init() are not fatal. > > In the cases where MPI_Init() returns MPI_ERR_RANK_FAIL_STOP to indicate a process failure, is the library usable or not? If the application can continue running through the failure, then the MPI library should still be usable, thus MPI_Init() must be fault tolerant in its initialization to be able to handle process failures. If the MPI implementation finds itself in trouble and cannot continue it should return MPI_ERR_CANNOT_CONTINUE from all subsequent calls including MPI_Init, if possible. > > > MPI_Finalize(): > --------------- > Problem: If a process fails before or during MPI_Finalize (and the error handler is not MPI_ERRORS_ARE_FATAL), what should this function return? Should that return value be consistent to all processes? > > To preserve locality of fault handling, a local process should not be explicitly forced to recognize the failure of a peer process that they never interact with neither directly (e.g., point-to-point) or indirectly (e.g., collective). So MPI_Finalize should be fault tolerant and keep trying to complete even in the presence of failures. > > MPI_Finalize is not required to be a collective operation, though it is often implemented that way. An implementation may need to delay the return from MPI_Finalize until its role in the failure information distribution channel is complete. But we should not require a multi-phase commit protocol to ensure that everyone either succeeds or returns some error. Implementations may do so internally in order to ensure that MPI_Finalize does not hang. > > If MPI_Finalize returns an error (say MPI_ERR_RANK_FAIL_STOP indicating a 'new to this rank' failure), what good is this information to the application? It cannot query for which rank(s) failed since MPI has been finalized. Nor can it initiate recovery. The best it could do is assume that all other processes failed and take local action. > > > MPI_Finalize: MPI_COMM_WORLD process rank 0: > -------------------------------------------- > In chapter 8, Example 8.7 illustrates that "Although it is not required that all processes return from MPI_Finalize, it is required that at least process 0 in MPI_COMM_WORLD return, so that users can know that the MPI portion of the computation is over." > > We deduced that the reasoning for this explanation was to allow for MPI implementation that create and destroy MPI processes during init/finalize from rank 0. Or worded differently, rank 0 is the only rank that can be assumed to exist before MPI_Init and after MPI_Finalize. > > Problem: So what if rank 0 fails at some point during the computation (or just some point during MPI_Finalize)? > > In the proposal, I added an advice to users to tell them to not depend on any specific ranks to exist before MPI_Init or after MPI_Finalize. So, in a faulty environment, the example will produce incorrect results under certain failure scenarios (e.g., failure of rank 0). > > In an MPI environment that depends on rank 0 for process creation and destruction, the failure of rank 0 is (should be?) critical and the MPI implementation will either abort the job or return MPI_ERR_CANNOT_CONTINUE from all calls to the MPI implementation. So we believe that the advice to users was a sufficient addition to this section. What do others think? > > > So MPI_Init seems to be a more complex issue than MPI_Finalize. What do folks think about the presented problems and possible solutions? Are there other issues not mentioned here that we should be addressing? > > -- Josh > > Run-Through Stabilization Proposal: > https://*svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilization > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://*www.*cs.indiana.edu/~jjhursey > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

[Mpi3-ft] MPI_Init / MPI_Finalize
by Joshua Hursey 25 Aug '10

25 Aug '10

During the discussion of the run-though stabilization proposal today on the teleconf, we spent a while discussing the expected behavior of MPI_Init and MPI_Finalize in the presence of process failures. I would like to broaden the discussion a bit to help pin down the expected behavior. MPI_Init(): ----------- Problem: If a process fails before or during MPI_Init, what should the MPI implementation do? The current standard says nothing about the return value of MPI_Init() (Ch. 8.7). To the greatest possible extent the application should not be put in danger if it wishes to ignore errors (assumes MPI_ERRORS_ARE_FATAL), so returning an error from this function (in contrast to aborting the job) might be dangerous. However, if the application is prepared to handle process failures, it is unable to communicate that information to the MPI implementation until after the completion of MPI_Init(). So a couple of solutions were presented each with pros and cons (please fill in if I missed any): 1) If a process fails in MPI_Init() (default error handler is MPI_ERRORS_ARE_FATAL) then the entire job is aborted (similar to calling MPI_Abort on MPI_COMM_WORLD). 2) If a process fails in MPI_Init() the MPI implementation will return an appropriate error code/class (e.g., MPI_ERR_RANK_FAIL_STOP), and all subsequent calls into the MPI implementation will return the error class MPI_ERR_OTHER (should be create a MPI_ERR_NOT_ACTIVE?). Applications should eventually notice the error and terminate. 3) Allow the application to register only the MPI_ERRORS_RETURN handle on MPI_COMM_WORLD before MPI_Init() using the MPI_Errhandler_set() function. Errors that occur before the MPI_Errhandler_set() call are fatal. Errors afterward, including during MPI_Init() are not fatal. In the cases where MPI_Init() returns MPI_ERR_RANK_FAIL_STOP to indicate a process failure, is the library usable or not? If the application can continue running through the failure, then the MPI library should still be usable, thus MPI_Init() must be fault tolerant in its initialization to be able to handle process failures. If the MPI implementation finds itself in trouble and cannot continue it should return MPI_ERR_CANNOT_CONTINUE from all subsequent calls including MPI_Init, if possible. MPI_Finalize(): --------------- Problem: If a process fails before or during MPI_Finalize (and the error handler is not MPI_ERRORS_ARE_FATAL), what should this function return? Should that return value be consistent to all processes? To preserve locality of fault handling, a local process should not be explicitly forced to recognize the failure of a peer process that they never interact with neither directly (e.g., point-to-point) or indirectly (e.g., collective). So MPI_Finalize should be fault tolerant and keep trying to complete even in the presence of failures. MPI_Finalize is not required to be a collective operation, though it is often implemented that way. An implementation may need to delay the return from MPI_Finalize until its role in the failure information distribution channel is complete. But we should not require a multi-phase commit protocol to ensure that everyone either succeeds or returns some error. Implementations may do so internally in order to ensure that MPI_Finalize does not hang. If MPI_Finalize returns an error (say MPI_ERR_RANK_FAIL_STOP indicating a 'new to this rank' failure), what good is this information to the application? It cannot query for which rank(s) failed since MPI has been finalized. Nor can it initiate recovery. The best it could do is assume that all other processes failed and take local action. MPI_Finalize: MPI_COMM_WORLD process rank 0: -------------------------------------------- In chapter 8, Example 8.7 illustrates that "Although it is not required that all processes return from MPI_Finalize, it is required that at least process 0 in MPI_COMM_WORLD return, so that users can know that the MPI portion of the computation is over." We deduced that the reasoning for this explanation was to allow for MPI implementation that create and destroy MPI processes during init/finalize from rank 0. Or worded differently, rank 0 is the only rank that can be assumed to exist before MPI_Init and after MPI_Finalize. Problem: So what if rank 0 fails at some point during the computation (or just some point during MPI_Finalize)? In the proposal, I added an advice to users to tell them to not depend on any specific ranks to exist before MPI_Init or after MPI_Finalize. So, in a faulty environment, the example will produce incorrect results under certain failure scenarios (e.g., failure of rank 0). In an MPI environment that depends on rank 0 for process creation and destruction, the failure of rank 0 is (should be?) critical and the MPI implementation will either abort the job or return MPI_ERR_CANNOT_CONTINUE from all calls to the MPI implementation. So we believe that the advice to users was a sufficient addition to this section. What do others think? So MPI_Init seems to be a more complex issue than MPI_Finalize. What do folks think about the presented problems and possible solutions? Are there other issues not mentioned here that we should be addressing? -- Josh Run-Through Stabilization Proposal: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

[Mpi3-ft] FT Working Group Teleconf
by Joshua Hursey 24 Aug '10

24 Aug '10

Reminder that we have a teleconf tomorrow, Wed 8/25/2010, at 12 noon EDT. Date: August 25, 2010 Time: 12 noon EDT/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Piggybacking (Greg B., George B., Darius B) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Nonblocking Process Management (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Async-proc-mgmt - Run-through stabilization proposal introduction (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - other items? Respond to the list if you have any additional topics that you would like to discuss. -- Josh ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

[Mpi3-ft] Run-Through Stabilization
by Joshua Hursey 24 Aug '10

24 Aug '10

I recently started to generate a proposal to support applications that wish to run-through process fail-stop failures. This is an attempt to pull apart the current FT proposals into two camps: stabilization, and recovery. The run-through stabilization proposal is meant to provide the foundation for and a complement to an eventual recovery proposal. The run-through stabilization proposal aggregates much of the discussion in the working group so far regarding error management and validation of communicators. I modified some of the interfaces based on some recent experimentation with the interfaces. I have also been reading through various parts of the standard to find semantics, interfaces, and discussions that will need to be modified in order to support run-though stabilization. The current draft on the wiki is rough and in development. I have a few more notes and changes to make, but I wanted to circulate the current state before the teleconf tomorrow. This way I can introduce the proposal and we can start discussing some of the concepts. My hope is that this proposal helps spark some more discussion in the working group. The proposal is linked off of the main FT working group wiki, and is at the link below: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… Concurrently with the development of the proposal text, I am working on a prototype implementation in Open MPI that should help guide the process of refining the proposal. -- Josh ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Question about MPI_ANY_SOURCE and process failures
by Fab Tillier 19 Aug '10

19 Aug '10

Joshua Hursey wrote on Thu, 19 Aug 2010 at 12:09:54 > I think it should be if ANY process fails on the communicator then it > should return an error. > > I don't think it should be ALL, per your comments below. Additionally, > ALL is not workable even if we do {ALL-self} since the intention of the > receive is to receive one message, and the MPI interface cannot assume > that the remaining processes will ever provide a message to the > application. > > NEVER is probably not what we want either for the same reason. If only > a subset of the processes in the communicator will ever send a message > (due to app. design) the MPI library does not know if the procs that > have failed are important or not. > > So I think the ANY process fail option is the only one that makes sense > here. Erez had documented much of the behavior for this in his error reporting rules available here: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/error_report_rules For a receive from source = MPI_ANY_SOURCE, see example E2 - the MPI_Wait fails if any error is detected on the communicator. E3 is similar, but shows the blocking receive use case. > The state of the 'status' object should point to one of the process > failures. If there are concurrent failures it is hard to tell which was > first, so the semantics should probably just say that any one of the > failed processes will be identified. Then the user should use a > 'validate' command to figure out which ones have failed. > > -- Josh > > P.S. I am starting to work on a slightly more formal fail-though > proposal for the group. This separates the interface/semantic issues for > fail-through from recovery. This will help get us through some of the > broader issues of stability before complicating the discussion with > (multiple, concurrent) recoveries. More on this in the next couple weeks. I don't know how much the rules Erez put together would apply to this work, but I'd suspect it would serve as a good starting point. -Fab > On Aug 19, 2010, at 2:46 PM, Solt, David George wrote: > >> err = MPI_Recv(....., rankX, ..., comm, status); >> >> if communication to rankX fails, this receive will return with err. >> >> Err = MPI_Recv(...., MPI_ANY_SOURCE, ..., comm, status); >> >> When does this MPI_Recv return a failure? When ANY rank in comm is > unreachable or when ALL ranks in comm are unreachable. Since self is > always reachable, the ALL option is really NEVER. >> >> We had been assuming ALL/NEVER but will likely change to ANY. In > such case, status points to the first failed rank that could have > matched the request. >> >> Thanks, >> Dave >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0