January 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] simplified FT proposal
by Josh Hursey 17 Jan '12

17 Jan '12

The next teleconf is tomorrow at noon. I'm sendin gout the announcement shortly. -- Josh On Tue, Jan 17, 2012 at 4:21 PM, Anthony Skjellum <tony(a)cis.uab.edu> wrote: > Josh, I just sent you a note, agreeing to do a writeup. I need a few > days. When is next telecon? > > Tony > > > ----- Original Message ----- > From: "Josh Hursey" <jjhursey(a)open-mpi.org> > To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" < > mpi3-ft(a)lists.mpi-forum.org> > Sent: Tuesday, January 17, 2012 3:15:13 PM > Subject: Re: [Mpi3-ft] simplified FT proposal > > > Tony, > > > I have to admit that I am getting a bit lost in your multipart > presentation of your proposal. So that is making it difficult for me to > reason about it as a whole. Can you synthesis your proposal (either as a > separate email thread or on the wiki) and maybe we can discuss it further > on the teleconf and mailing list? That will give us a single place to point > to and refine during the subsequent discussions. > > > Thanks, > Josh > > > On Mon, Jan 16, 2012 at 4:18 PM, Anthony Skjellum < > tony(a)runtimecomputing.com > wrote: > > > > More > > > #5) > > > A parallel thread, running on the same group as the communicator used for > the user program in FTBLOCK, would be free to be running a gossip protocol, > instantiated > a) by the user at his/her choice > b) by MPI at its choice, by user demand > > > This parallel inspector thread propagates error state, and accepts input > to the error state by allowing user programs to assert error state within > the FTBLOCK. > > > You can think of this parallel thread as cooperation of the progress > engines with gossip, or as a completely optional activity, created by the > user, or as > a part of the FT proposal. As long as there are local ways to add error > state, we can write this as a layered library for the sake of making sure > that local > errors are propagated (remembering that MPI is returning local errors to > the user inside his/her FTBLOCK). > > > If we want the implementation to provide/support this kind of FT-friendly > error propagation, that is a bigger step, which I am not advocating as > required. > I think it would be nice to have this be part of a fault tolerant > communication space. But, I am OK with users doing this for themselves, > esp. because they > can control granularity better. > > > #6) I think MPI_COMM_WORLD is a bummer in the FT world. We don't want it > to hang around very long after we get going. If we are really working on > subgroups, and these subgroups form a hierarchical graph, rather than an > all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD > after > the first process breaks. So - as an example of pain of MPI scalability - > the build-down model of MPI-1 is less good for this than a build-up > model, where we find a way for groups to rendezvous. Obviously, for > convenience, the all-to-all virtual topology in the onset of MPI_Init() is > nice, > but I am assuming that errors may happen quite quickly. > > > MPI_Init() with no failures is only good for a limited window of time, > given failure rates. During this time, we would like to "turn off > MPI_COMM_WORLD" > unless we really don't need it, or at least not have to reconstruct it > ever if we don't need it. > > > So, we need to agree on a fault-friendly MPI_Init()... > > > An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of > spawning MPI, or the spawn command that creates the world, create an > effective communicator. We should generalize MPI_Init() to support outcomes > other than pure success, such as partial success (e.g., smaller world than > stipulated). > > > A really good approach would be to only create the actual communicators > you really need to run your process, not the virtual-all-to-all world of > MPI-1 and MPI-2. > But, that is a heresy, so forgive me. > > > Tony > > > > > > > > > > > > > > > > On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum < > tony(a)runtimecomputing.com > wrote: > > > Satantan, > > > Here are my further thoughts, which hopefully includes an answer to your > questions :-) > > > 1) Everything is set to "ERRORS return"; the goal is to get local errors > back if they are available. > > > 2) I would emphasize non-blocking operations, but blocking operations > implemented with internal timeout could return a timeout-type error > > > 3) You don't have to return the same return code, or the same results in > all processes in the communicator, you can get erroneous results > or local failures; the functions are also allowed to produce incorrect > results [and we should then discuss what error reporting means here... > I am happy with local errors returned where known, recognizing those > processes may die before the bottom of the block. However, I also > expect the implementation to do its best to propagate error state > knowledge within this FTBLOCK, based organically on ongoing communication > or on gossip if an implementation so chose.] > > > Also, because we assume that there is also algorithmic fault tolerance at > work, local errors may be raised by the application because it is doing > checks > for validity etc. > > > So, either MPI or the application may raise local errors prior to the > bottom of the FTBLOCK, and the success of the bottom of the block must be > allowed > to be fail, just based on ABFT inputs from the application to MPI, not > just based on MPI's opinion. > > > 4) If you are willing to do everything nonblocking, then I can describe > the test at the bottom of the FTBLOCK as follows: > > > The test operation at the bottom of the FTBLOCK is effectively a > generalized WAIT_ALL, that completes or doesn't complete all the outstanding > requests, returning errors related to the faults observed, and providing a > unified 0/1 success failure state consistently across the group of comm > [or surviving members thereof]. > > > In my view, the application as well as MPI can contribute error state as > input to the FTBLOCK test. > > > Also, the application that gets local errors inside the loop already is > immediately ready to punt at that point, and do a jump to the bottom of the > loop. Let's > assume it is required to do all the operations before getting to the > bottom of the loop, for now, and we just allow that some of these may > return further > errors (I am trying to keep it simple), and MPI-like rules of all > attempting the operations. If we can get this to work, we can weaken it > later. > > > There is no good way to describe some BLOCKING, some nonblocking > operations, because we have no descriptor to tell us if something failed > that previously > returned... and did not give a local error, so I am not going to pursue > BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this later, > if we can get a consistent solution using all nonblocking. > > > Please tell me what you think. > > > Thanks for responding! > > > Tony > > > > > > > On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan < sayantan.sur(a)intel.com> wrote: > > > > > > > Hi Tony, > > > > In the example semantics you mentioned, are the “ops” required to return > the same result on all processors? Although this doesn’t change the API > “op”, but it does change completion semantics of almost all MPI ops. I hope > I am correctly interpreting your message. > > > > > Thanks. > > > > === > > Sayantan Sur, Ph.D. > > Intel Corp. > > > > > > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of Anthony Skjellum > Sent: Sunday, January 15, 2012 7:06 PM > > > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] simplified FT proposal > > > > > > > > Everyone, I think we need to start from scratch. > > > > > > We should look for minimal fault-tolerant models that are achievable and > useful. They may allow for a combination of faults (process and network), > but in the end, as discussed in San Jose: > > > > > > FTBLOCK > > > -------------- > > > Start_Block(comm) > > > > > > op [normal MPI operation on communicator specified by Start_Block, or > subset thereof] > > > op > > > op > > > > > > Test_Block(comm) > > > > > > Which either succeeds, or fails, on the whole list of operations, followed > by ways to reconstruct communicators and add back processes (easily), > > > provides for 3-level fault tolerant model > > > > > > a) Simply retry if the kind of error at the Test_Block is retryable > > > b) Simply reconstruct the communicator, use algorithmic fault tolerance to > get lost data, and retry the block > > > c) drop back to 1 or more levels of checkpoint-restart. > > > > > > We can envision in this model an unrolling of work, in terms of a > parameter N, if there is a lot of vector work, to allow granularity control > > > as a function of fault environment. > > > > > > In some sense, a simpler model, that provides for efforts to detect > > > a) by MPI > > > b) allowing application monitor to assert failure asynchronously to this > loop > > > > > > provides a more general ability to have coverage of faults, including but > not limited to process faults and possible network faults. > > > > > > It changes the API very little. > > > > > > It motivates the use of buffers, not zero copy, to support the fact that > you may to roll-back a series of operations, thereby revealing fault-free > overhead directly. > > > > > > Start_Block and Test_Block are collective and synchronizing, such as > barriers. Because we have uncertainty to within a message, multiple > barriers (as mentioned by George Bosilca K to me in a sidebar at the > meeting). > > > > > > We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we > can. Once we have this stable intermediate form, we explore more > > > options. > > > > > > I think it is important to recognize that the reconstruction step, > including readding processes and making new communicators may mean smarter > Join operations. It is clear we need to be able to treat failures during > the recovery process, and use a second level loop, possibly bombing out > > > to checkpoint, if we cannot make net progress on recovery because of > unmodeled error issues. > > > > > > The testing part leverages all the learning so far, but needn't be > restricted to modeled errors like process faults. There can be modeled and > unmodeled faults. Based on what fault comes up, the user application then > has to decide how hard a retry to do, whether just to add processes, > > > whether just to retry the loop, whether to go to a checkpoint, whether to > restart the APP. MPI could give advice, based on its understanding of > > > the fault model, in terms of sufficient conditions for "working harder" vs > "trying the easiest" for fault models it understands somewhat for a system. > > > > > > Now, the comments here are a synopsis of part of the side bars and open > discussion we had in San Jose, distilled a bit. I want to know why > > > we can't start with this, succeed with this, implement and test it, and > having succeeded, do more in a future 3.y, y > x release, given user > experience. > > > > > > I am not speaking to the choice of "killing all communicators" as with > FT-MPI, or "just remaking those you need to remake." I think we need to > resolve. Honestly, groups own the fault property, not communicators, and > all groups held by communicators where the fault happened should be > rebuilt, not all communicators... Let's argue on that. > > > > > > So, my suggestion is REBOOT the proposal with something along lines above, > unless you all see this is no better. > > > > > > With kind regards, > > > Tony > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan < sayantan.sur(a)intel.com > > wrote: > > > > Hi Bill, > > > > I am in agreement with your suggestion to have a collective over a > communicator that returns a new communicator containing ranks “alive some > point during construction”. It provides cleaner semantics. The example was > merely trying to utilize the new MPI_Comm_create_group API that the Forum > is considering. > > > > MPI_Comm_check provides a method to form global consensus in that all > ranks in comm did call it. It does not imply anything about current status > of comm, or even the status “just before” the call returns. During the > interval before the next call to MPI_Comm_check, it is possible that many > ranks fail. However, the app/lib using MPI knows the point where everyone > was alive. > > > > Thanks. > > > > > > === > > Sayantan Sur, Ph.D. > > Intel Corp. > > > > > > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of William Gropp > Sent: Sunday, January 15, 2012 2:41 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] simplified FT proposal > > > > > > One concern that I have with fault tolerant proposals has to do with races > in the specification. This is an area where users often "just want it to > work" but getting it right is tricky. In the example here, the > "alive_group" is really only that at some moment shortly before > "MPI_Comm_check" returns (and possibly not even that). After that, it is > really the "group_of_processes_that_was_alive_at_some_point_in_the_past". > Since there are sometimes correlations in failures, this could happen even > if the initial failure is rare. An alternate form might be to have a > routine, collective over a communicator, that returns a new communicator > meeting some definition of "members were alive at some point during > construction". It wouldn't guarantee you could use it, but it would have > cleaner semantics. > > > > > > > Bill > > > > > > > On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote: > > > > > I would like to argue for a simplified version of the proposal that covers > a large percentage of use-cases and resists adding new “features” for the > full-range of ABFT techniques. It is good if we have a more pragmatic view > and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps > just want to do something like this: > > > > > > for(… really long time …) { > > > MPI_Comm_check(work_comm, &is_ok, &alive_group); > > > if(!is_ok) { > > > MPI_Comm_create_group(alive_group, …, &new_comm); > > > // re-balance workload and use new_comm in rest of computation > > > MPI_Comm_free(work_comm); // get rid of old comm > > > work_comm = new_comm; > > > } else { > > > // continue computation using work_comm > > > // if some proc failed in this iteration, roll back work done in this > iteration, go back to loop > > > } > > > } > > > > > > > > > > William Gropp > > > Director, Parallel Computing Institute > > > Deputy Director for Research > > > Institute for Advanced Computing Applications and Technologies > > > Paul and Cynthia Saylor Professor of Computer Science > > > University of Illinois Urbana-Champaign > > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > > > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > > > > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > Anthony Skjellum, PhD > Professor and Chair > Dept. of Computer and Information Sciences > Director, UAB Center for Information Assurance and Joint Forensic Research > ("The Center") > University of Alabama at Birmingham > +1-(205)934-8657; FAX: +1- (205)934-5473 > > ___________________________________________ > CONFIDENTIALITY: This e-mail and any attachments are confidential and > may be privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person, > use it for any purpose or store or copy the information in any medium. > > Please consider the environment before printing this e-mail > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] simplified FT proposal
by Anthony Skjellum 17 Jan '12

17 Jan '12

Josh, I just sent you a note, agreeing to do a writeup. I need a few days. When is next telecon? Tony ----- Original Message ----- From: "Josh Hursey" <jjhursey(a)open-mpi.org> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Sent: Tuesday, January 17, 2012 3:15:13 PM Subject: Re: [Mpi3-ft] simplified FT proposal Tony, I have to admit that I am getting a bit lost in your multipart presentation of your proposal. So that is making it difficult for me to reason about it as a whole. Can you synthesis your proposal (either as a separate email thread or on the wiki) and maybe we can discuss it further on the teleconf and mailing list? That will give us a single place to point to and refine during the subsequent discussions. Thanks, Josh On Mon, Jan 16, 2012 at 4:18 PM, Anthony Skjellum < tony(a)runtimecomputing.com > wrote: More #5) A parallel thread, running on the same group as the communicator used for the user program in FTBLOCK, would be free to be running a gossip protocol, instantiated a) by the user at his/her choice b) by MPI at its choice, by user demand This parallel inspector thread propagates error state, and accepts input to the error state by allowing user programs to assert error state within the FTBLOCK. You can think of this parallel thread as cooperation of the progress engines with gossip, or as a completely optional activity, created by the user, or as a part of the FT proposal. As long as there are local ways to add error state, we can write this as a layered library for the sake of making sure that local errors are propagated (remembering that MPI is returning local errors to the user inside his/her FTBLOCK). If we want the implementation to provide/support this kind of FT-friendly error propagation, that is a bigger step, which I am not advocating as required. I think it would be nice to have this be part of a fault tolerant communication space. But, I am OK with users doing this for themselves, esp. because they can control granularity better. #6) I think MPI_COMM_WORLD is a bummer in the FT world. We don't want it to hang around very long after we get going. If we are really working on subgroups, and these subgroups form a hierarchical graph, rather than an all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD after the first process breaks. So - as an example of pain of MPI scalability - the build-down model of MPI-1 is less good for this than a build-up model, where we find a way for groups to rendezvous. Obviously, for convenience, the all-to-all virtual topology in the onset of MPI_Init() is nice, but I am assuming that errors may happen quite quickly. MPI_Init() with no failures is only good for a limited window of time, given failure rates. During this time, we would like to "turn off MPI_COMM_WORLD" unless we really don't need it, or at least not have to reconstruct it ever if we don't need it. So, we need to agree on a fault-friendly MPI_Init()... An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of spawning MPI, or the spawn command that creates the world, create an effective communicator. We should generalize MPI_Init() to support outcomes other than pure success, such as partial success (e.g., smaller world than stipulated). A really good approach would be to only create the actual communicators you really need to run your process, not the virtual-all-to-all world of MPI-1 and MPI-2. But, that is a heresy, so forgive me. Tony On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum < tony(a)runtimecomputing.com > wrote: Satantan, Here are my further thoughts, which hopefully includes an answer to your questions :-) 1) Everything is set to "ERRORS return"; the goal is to get local errors back if they are available. 2) I would emphasize non-blocking operations, but blocking operations implemented with internal timeout could return a timeout-type error 3) You don't have to return the same return code, or the same results in all processes in the communicator, you can get erroneous results or local failures; the functions are also allowed to produce incorrect results [and we should then discuss what error reporting means here... I am happy with local errors returned where known, recognizing those processes may die before the bottom of the block. However, I also expect the implementation to do its best to propagate error state knowledge within this FTBLOCK, based organically on ongoing communication or on gossip if an implementation so chose.] Also, because we assume that there is also algorithmic fault tolerance at work, local errors may be raised by the application because it is doing checks for validity etc. So, either MPI or the application may raise local errors prior to the bottom of the FTBLOCK, and the success of the bottom of the block must be allowed to be fail, just based on ABFT inputs from the application to MPI, not just based on MPI's opinion. 4) If you are willing to do everything nonblocking, then I can describe the test at the bottom of the FTBLOCK as follows: The test operation at the bottom of the FTBLOCK is effectively a generalized WAIT_ALL, that completes or doesn't complete all the outstanding requests, returning errors related to the faults observed, and providing a unified 0/1 success failure state consistently across the group of comm [or surviving members thereof]. In my view, the application as well as MPI can contribute error state as input to the FTBLOCK test. Also, the application that gets local errors inside the loop already is immediately ready to punt at that point, and do a jump to the bottom of the loop. Let's assume it is required to do all the operations before getting to the bottom of the loop, for now, and we just allow that some of these may return further errors (I am trying to keep it simple), and MPI-like rules of all attempting the operations. If we can get this to work, we can weaken it later. There is no good way to describe some BLOCKING, some nonblocking operations, because we have no descriptor to tell us if something failed that previously returned... and did not give a local error, so I am not going to pursue BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this later, if we can get a consistent solution using all nonblocking. Please tell me what you think. Thanks for responding! Tony On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan < sayantan.sur(a)intel.com > wrote: Hi Tony, In the example semantics you mentioned, are the “ops” required to return the same result on all processors? Although this doesn’t change the API “op”, but it does change completion semantics of almost all MPI ops. I hope I am correctly interpreting your message. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of Anthony Skjellum Sent: Sunday, January 15, 2012 7:06 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal Everyone, I think we need to start from scratch. We should look for minimal fault-tolerant models that are achievable and useful. They may allow for a combination of faults (process and network), but in the end, as discussed in San Jose: FTBLOCK -------------- Start_Block(comm) op [normal MPI operation on communicator specified by Start_Block, or subset thereof] op op Test_Block(comm) Which either succeeds, or fails, on the whole list of operations, followed by ways to reconstruct communicators and add back processes (easily), provides for 3-level fault tolerant model a) Simply retry if the kind of error at the Test_Block is retryable b) Simply reconstruct the communicator, use algorithmic fault tolerance to get lost data, and retry the block c) drop back to 1 or more levels of checkpoint-restart. We can envision in this model an unrolling of work, in terms of a parameter N, if there is a lot of vector work, to allow granularity control as a function of fault environment. In some sense, a simpler model, that provides for efforts to detect a) by MPI b) allowing application monitor to assert failure asynchronously to this loop provides a more general ability to have coverage of faults, including but not limited to process faults and possible network faults. It changes the API very little. It motivates the use of buffers, not zero copy, to support the fact that you may to roll-back a series of operations, thereby revealing fault-free overhead directly. Start_Block and Test_Block are collective and synchronizing, such as barriers. Because we have uncertainty to within a message, multiple barriers (as mentioned by George Bosilca K to me in a sidebar at the meeting). We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we can. Once we have this stable intermediate form, we explore more options. I think it is important to recognize that the reconstruction step, including readding processes and making new communicators may mean smarter Join operations. It is clear we need to be able to treat failures during the recovery process, and use a second level loop, possibly bombing out to checkpoint, if we cannot make net progress on recovery because of unmodeled error issues. The testing part leverages all the learning so far, but needn't be restricted to modeled errors like process faults. There can be modeled and unmodeled faults. Based on what fault comes up, the user application then has to decide how hard a retry to do, whether just to add processes, whether just to retry the loop, whether to go to a checkpoint, whether to restart the APP. MPI could give advice, based on its understanding of the fault model, in terms of sufficient conditions for "working harder" vs "trying the easiest" for fault models it understands somewhat for a system. Now, the comments here are a synopsis of part of the side bars and open discussion we had in San Jose, distilled a bit. I want to know why we can't start with this, succeed with this, implement and test it, and having succeeded, do more in a future 3.y, y > x release, given user experience. I am not speaking to the choice of "killing all communicators" as with FT-MPI, or "just remaking those you need to remake." I think we need to resolve. Honestly, groups own the fault property, not communicators, and all groups held by communicators where the fault happened should be rebuilt, not all communicators... Let's argue on that. So, my suggestion is REBOOT the proposal with something along lines above, unless you all see this is no better. With kind regards, Tony On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan < sayantan.sur(a)intel.com > wrote: Hi Bill, I am in agreement with your suggestion to have a collective over a communicator that returns a new communicator containing ranks “alive some point during construction”. It provides cleaner semantics. The example was merely trying to utilize the new MPI_Comm_create_group API that the Forum is considering. MPI_Comm_check provides a method to form global consensus in that all ranks in comm did call it. It does not imply anything about current status of comm, or even the status “just before” the call returns. During the interval before the next call to MPI_Comm_check, it is possible that many ranks fail. However, the app/lib using MPI knows the point where everyone was alive. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: mpi3-ft-bounces(a)lists.mpi-forum.org ] On Behalf Of William Gropp Sent: Sunday, January 15, 2012 2:41 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal One concern that I have with fault tolerant proposals has to do with races in the specification. This is an area where users often "just want it to work" but getting it right is tricky. In the example here, the "alive_group" is really only that at some moment shortly before "MPI_Comm_check" returns (and possibly not even that). After that, it is really the "group_of_processes_that_was_alive_at_some_point_in_the_past". Since there are sometimes correlations in failures, this could happen even if the initial failure is rare. An alternate form might be to have a routine, collective over a communicator, that returns a new communicator meeting some definition of "members were alive at some point during construction". It wouldn't guarantee you could use it, but it would have cleaner semantics. Bill On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote: I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and resists adding new “features” for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this: for(… really long time …) { MPI_Comm_check(work_comm, &is_ok, &alive_group); if(!is_ok) { MPI_Comm_create_group(alive_group, …, &new_comm); // re-balance workload and use new_comm in rest of computation MPI_Comm_free(work_comm); // get rid of old comm work_comm = new_comm; } else { // continue computation using work_comm // if some proc failed in this iteration, roll back work done in this iteration, go back to loop } } William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensic Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

Re: [Mpi3-ft] simplified FT proposal
by Josh Hursey 17 Jan '12

17 Jan '12

Tony, I have to admit that I am getting a bit lost in your multipart presentation of your proposal. So that is making it difficult for me to reason about it as a whole. Can you synthesis your proposal (either as a separate email thread or on the wiki) and maybe we can discuss it further on the teleconf and mailing list? That will give us a single place to point to and refine during the subsequent discussions. Thanks, Josh On Mon, Jan 16, 2012 at 4:18 PM, Anthony Skjellum <tony(a)runtimecomputing.com > wrote: > More > > #5) > > A parallel thread, running on the same group as the communicator used for > the user program in FTBLOCK, would be free to be running a gossip protocol, > instantiated > a) by the user at his/her choice > b) by MPI at its choice, by user demand > > This parallel inspector thread propagates error state, and accepts input > to the error state by allowing user programs to assert error state within > the FTBLOCK. > > You can think of this parallel thread as cooperation of the progress > engines with gossip, or as a completely optional activity, created by the > user, or as > a part of the FT proposal. As long as there are local ways to add error > state, we can write this as a layered library for the sake of making sure > that local > errors are propagated (remembering that MPI is returning local errors to > the user inside his/her FTBLOCK). > > If we want the implementation to provide/support this kind of FT-friendly > error propagation, that is a bigger step, which I am not advocating as > required. > I think it would be nice to have this be part of a fault tolerant > communication space. But, I am OK with users doing this for themselves, > esp. because they > can control granularity better. > > #6) I think MPI_COMM_WORLD is a bummer in the FT world. We don't want it > to hang around very long after we get going. If we are really working on > subgroups, and these subgroups form a hierarchical graph, rather than an > all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD > after > the first process breaks. So - as an example of pain of MPI scalability - > the build-down model of MPI-1 is less good for this than a build-up > model, where we find a way for groups to rendezvous. Obviously, for > convenience, the all-to-all virtual topology in the onset of MPI_Init() is > nice, > but I am assuming that errors may happen quite quickly. > > MPI_Init() with no failures is only good for a limited window of time, > given failure rates. During this time, we would like to "turn off > MPI_COMM_WORLD" > unless we really don't need it, or at least not have to reconstruct it > ever if we don't need it. > > So, we need to agree on a fault-friendly MPI_Init()... > > An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of > spawning MPI, or the spawn command that creates the world, create an > effective communicator. We should generalize MPI_Init() to support > outcomes other than pure success, such as partial success (e.g., smaller > world than stipulated). > > A really good approach would be to only create the actual communicators > you really need to run your process, not the virtual-all-to-all world of > MPI-1 and MPI-2. > But, that is a heresy, so forgive me. > > Tony > > > > > > > > On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum < > tony(a)runtimecomputing.com> wrote: > >> Satantan, >> >> Here are my further thoughts, which hopefully includes an answer to your >> questions :-) >> >> 1) Everything is set to "ERRORS return"; the goal is to get local errors >> back if they are available. >> >> 2) I would emphasize non-blocking operations, but blocking operations >> implemented with internal timeout could return a timeout-type error >> >> 3) You don't have to return the same return code, or the same results in >> all processes in the communicator, you can get erroneous results >> or local failures; the functions are also allowed to produce >> incorrect results [and we should then discuss what error reporting means >> here... >> I am happy with local errors returned where known, recognizing those >> processes may die before the bottom of the block. However, I also >> expect the implementation to do its best to propagate error state >> knowledge within this FTBLOCK, based organically on ongoing communication >> or on gossip if an implementation so chose.] >> >> Also, because we assume that there is also algorithmic fault tolerance >> at work, local errors may be raised by the application because it is doing >> checks >> for validity etc. >> >> So, either MPI or the application may raise local errors prior to the >> bottom of the FTBLOCK, and the success of the bottom of the block must be >> allowed >> to be fail, just based on ABFT inputs from the application to MPI, not >> just based on MPI's opinion. >> >> 4) If you are willing to do everything nonblocking, then I can describe >> the test at the bottom of the FTBLOCK as follows: >> >> The test operation at the bottom of the FTBLOCK is effectively a >> generalized WAIT_ALL, that completes or doesn't complete all the outstanding >> requests, returning errors related to the faults observed, and providing >> a unified 0/1 success failure state consistently across the group of comm >> [or surviving members thereof]. >> >> In my view, the application as well as MPI can contribute error state as >> input to the FTBLOCK test. >> >> Also, the application that gets local errors inside the loop already is >> immediately ready to punt at that point, and do a jump to the bottom of the >> loop. Let's >> assume it is required to do all the operations before getting to the >> bottom of the loop, for now, and we just allow that some of these may >> return further >> errors (I am trying to keep it simple), and MPI-like rules of all >> attempting the operations. If we can get this to work, we can weaken it >> later. >> >> There is no good way to describe some BLOCKING, some nonblocking >> operations, because we have no descriptor to tell us if something failed >> that previously >> returned... and did not give a local error, so I am not going to pursue >> BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this >> later, >> if we can get a consistent solution using all nonblocking. >> >> Please tell me what you think. >> >> Thanks for responding! >> >> Tony >> >> >> On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan <sayantan.sur(a)intel.com>wrote: >> >>> Hi Tony,**** >>> >>> ** ** >>> >>> In the example semantics you mentioned, are the “ops” required to return >>> the same result on all processors? Although this doesn’t change the API >>> “op”, but it does change completion semantics of almost all MPI ops. I hope >>> I am correctly interpreting your message.**** >>> >>> ** ** >>> >>> Thanks.**** >>> >>> ** ** >>> >>> ===**** >>> >>> Sayantan Sur, Ph.D.**** >>> >>> Intel Corp.**** >>> >>> ** ** >>> >>> *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: >>> mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *Anthony Skjellum >>> *Sent:* Sunday, January 15, 2012 7:06 PM >>> >>> *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> *Subject:* Re: [Mpi3-ft] simplified FT proposal**** >>> >>> ** ** >>> >>> Everyone, I think we need to start from scratch.**** >>> >>> ** ** >>> >>> We should look for minimal fault-tolerant models that are achievable and >>> useful. They may allow for a combination of faults (process and network), >>> but in the end, as discussed in San Jose:**** >>> >>> ** ** >>> >>> FTBLOCK**** >>> >>> --------------**** >>> >>> Start_Block(comm)**** >>> >>> ** ** >>> >>> op [normal MPI operation on communicator specified by Start_Block, or >>> subset thereof]**** >>> >>> op**** >>> >>> op**** >>> >>> ** ** >>> >>> Test_Block(comm)**** >>> >>> ** ** >>> >>> Which either succeeds, or fails, on the whole list of operations, >>> followed by ways to reconstruct communicators and add back processes >>> (easily),**** >>> >>> provides for 3-level fault tolerant model**** >>> >>> ** ** >>> >>> a) Simply retry if the kind of error at the Test_Block is retryable**** >>> >>> b) Simply reconstruct the communicator, use algorithmic fault tolerance >>> to get lost data, and retry the block**** >>> >>> c) drop back to 1 or more levels of checkpoint-restart.**** >>> >>> ** ** >>> >>> We can envision in this model an unrolling of work, in terms of a >>> parameter N, if there is a lot of vector work, to allow granularity control >>> **** >>> >>> as a function of fault environment.**** >>> >>> ** ** >>> >>> In some sense, a simpler model, that provides for efforts to detect**** >>> >>> a) by MPI**** >>> >>> b) allowing application monitor to assert failure asynchronously to this >>> loop**** >>> >>> ** ** >>> >>> provides a more general ability to have coverage of faults, including >>> but not limited to process faults and possible network faults.**** >>> >>> ** ** >>> >>> It changes the API very little.**** >>> >>> ** ** >>> >>> It motivates the use of buffers, not zero copy, to support the fact that >>> you may to roll-back a series of operations, thereby revealing fault-free >>> overhead directly.**** >>> >>> ** ** >>> >>> Start_Block and Test_Block are collective and synchronizing, such as >>> barriers. Because we have uncertainty to within a message, multiple >>> barriers (as mentioned by George Bosilca K to me in a sidebar at the >>> meeting).**** >>> >>> ** ** >>> >>> We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if >>> we can. Once we have this stable intermediate form, we explore more**** >>> >>> options.**** >>> >>> ** ** >>> >>> I think it is important to recognize that the reconstruction step, >>> including readding processes and making new communicators may mean smarter >>> Join operations. It is clear we need to be able to treat failures during >>> the recovery process, and use a second level loop, possibly bombing out* >>> *** >>> >>> to checkpoint, if we cannot make net progress on recovery because of >>> unmodeled error issues.**** >>> >>> ** ** >>> >>> The testing part leverages all the learning so far, but needn't be >>> restricted to modeled errors like process faults. There can be modeled and >>> unmodeled faults. Based on what fault comes up, the user application then >>> has to decide how hard a retry to do, whether just to add processes,**** >>> >>> whether just to retry the loop, whether to go to a checkpoint, whether >>> to restart the APP. MPI could give advice, based on its understanding of >>> **** >>> >>> the fault model, in terms of sufficient conditions for "working harder" >>> vs "trying the easiest" for fault models it understands somewhat for a >>> system.**** >>> >>> ** ** >>> >>> Now, the comments here are a synopsis of part of the side bars and open >>> discussion we had in San Jose, distilled a bit. I want to know why **** >>> >>> we can't start with this, succeed with this, implement and test it, and >>> having succeeded, do more in a future 3.y, y > x release, given user >>> experience.**** >>> >>> ** ** >>> >>> I am not speaking to the choice of "killing all communicators" as with >>> FT-MPI, or "just remaking those you need to remake." I think we need to >>> resolve. Honestly, groups own the fault property, not communicators, and >>> all groups held by communicators where the fault happened should be >>> rebuilt, not all communicators... Let's argue on that.**** >>> >>> ** ** >>> >>> So, my suggestion is REBOOT the proposal with something along lines >>> above, unless you all see this is no better.**** >>> >>> ** ** >>> >>> With kind regards,**** >>> >>> Tony**** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan <sayantan.sur(a)intel.com> >>> wrote:**** >>> >>> Hi Bill,**** >>> >>> **** >>> >>> I am in agreement with your suggestion to have a collective over a >>> communicator that returns a new communicator containing ranks “alive some >>> point during construction”. It provides cleaner semantics. The example was >>> merely trying to utilize the new MPI_Comm_create_group API that the Forum >>> is considering.**** >>> >>> **** >>> >>> MPI_Comm_check provides a method to form global consensus in that all >>> ranks in comm did call it. It does not imply anything about current status >>> of comm, or even the status “just before” the call returns. During the >>> interval before the next call to MPI_Comm_check, it is possible that many >>> ranks fail. However, the app/lib using MPI knows the point where everyone >>> was alive.**** >>> >>> **** >>> >>> Thanks.**** >>> >>> **** >>> >>> ===**** >>> >>> Sayantan Sur, Ph.D.**** >>> >>> Intel Corp.**** >>> >>> **** >>> >>> *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: >>> mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *William Gropp >>> *Sent:* Sunday, January 15, 2012 2:41 PM >>> *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> *Subject:* Re: [Mpi3-ft] simplified FT proposal**** >>> >>> **** >>> >>> One concern that I have with fault tolerant proposals has to do with >>> races in the specification. This is an area where users often "just want >>> it to work" but getting it right is tricky. In the example here, the >>> "alive_group" is really only that at some moment shortly before >>> "MPI_Comm_check" returns (and possibly not even that). After that, it is >>> really the "group_of_processes_that_was_alive_at_some_point_in_the_past". >>> Since there are sometimes correlations in failures, this could happen even >>> if the initial failure is rare. An alternate form might be to have a >>> routine, collective over a communicator, that returns a new communicator >>> meeting some definition of "members were alive at some point during >>> construction". It wouldn't guarantee you could use it, but it would have >>> cleaner semantics.**** >>> >>> **** >>> >>> Bill**** >>> >>> **** >>> >>> On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote:**** >>> >>> ** ** >>> >>> I would like to argue for a simplified version of the proposal that >>> covers a large percentage of use-cases and resists adding new “features” >>> for the full-range of ABFT techniques. It is good if we have a more >>> pragmatic view and not sacrifice the entire FT proposal for the 1% fringe >>> cases. Most apps just want to do something like this:**** >>> >>> **** >>> >>> for(… really long time …) {**** >>> >>> MPI_Comm_check(work_comm, &is_ok, &alive_group);**** >>> >>> if(!is_ok) {**** >>> >>> MPI_Comm_create_group(alive_group, …, &new_comm);**** >>> >>> // re-balance workload and use new_comm in rest of computation**** >>> >>> MPI_Comm_free(work_comm); // get rid of old comm**** >>> >>> work_comm = new_comm;**** >>> >>> } else {**** >>> >>> // continue computation using work_comm**** >>> >>> // if some proc failed in this iteration, roll back work done in >>> this iteration, go back to loop**** >>> >>> }**** >>> >>> }**** >>> >>> **** >>> >>> **** >>> >>> William Gropp**** >>> >>> Director, Parallel Computing Institute**** >>> >>> Deputy Director for Research**** >>> >>> Institute for Advanced Computing Applications and Technologies**** >>> >>> Paul and Cynthia Saylor Professor of Computer Science**** >>> >>> University of Illinois Urbana-Champaign**** >>> >>> **** >>> >>> **** >>> >>> **** >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft**** >>> >>> >>> >>> **** >>> >>> ** ** >>> >>> -- >>> Tony Skjellum, PhD >>> RunTime Computing Solutions, LLC >>> tony(a)runtimecomputing.com >>> direct: +1-205-314-3595 >>> cell: +1-205-807-4968**** >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> >> >> -- >> Tony Skjellum, PhD >> RunTime Computing Solutions, LLC >> tony(a)runtimecomputing.com >> direct: +1-205-314-3595 >> cell: +1-205-807-4968 >> >> > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] simplified FT proposal
by Josh Hursey 17 Jan '12

17 Jan '12

Sayantan, Thanks for presentation of an alternative proposal. Alternative proposals are invaluable in helping the group work towards a more refined final proposal for the forum. I do have some questions, comments, and concerns about your proposal detailed below by section. Thanks, Josh (1) Consider a communicator with 1 million processes and 10 die. Would you rather reason about 10 dead or (1M-10) alive? At some point it does not matter much, but for the common case where the number of failures is much less than the size of the MPI universe then it is computationally easier to reason about the set of failed. What is the exact meaning of "is_ok"? (2) The phrase "collectively active/inactive" is bad - as illustrated by the reaction in the meeting. For the discussion consider a MPI_Reduce(). The root may be connected to alive processes in a tree structure, and is waiting patiently for a result. Somewhere in the tree a process fails before entering the MPI_Reduce(). Is it valid for the root to wait forever - locally observed behavior is a hang? or must the processes in the tree propagate the error - locally observed behavior is a transitive error? So considering that situation what must we say in the MPI standard about collective operations? (3) I'm fine with removing most of the failure detector statement if that is what the group wants. It served its purpose as a method to ground the semantics. I feel it is useful to keep in (it is precise and well grounded in theory), but if it has become distracting then we can consider removing it. I'm concerned with the language that "requests may complete with MPI_ERR_PROC_FAIL_STOP" as a way to allude to the option of MPI being able to delay notification. The wording also leaves open the opportunity for an implementation to wait forever to deliver the failure notification thus hanging the program. That is one of the nice aspects of stating explicitly that eventually every process -may- know of the failure, and that communication will complete (as stated in each of the subchapters). But that conversation tends to come back around to well we should make an explicit statement like we have in the document now... (4) Yeah I think Failure Handlers are distracting the conversation, and we should consider bringing them in separately. It would be useful to have a notification of process failure -not- tied to the current MPI call context. Which is what the failure handlers were meant to do. However the following question is what can a user do inside a failure handler, which is what lead to the discussion of the different modes, calling order, ... There is a strong use case for these from UTK to which they can elaborate better than I. (5) So you are proposing the MPI_ANY_SOURCE receives -never- return in error, even if the only process alive in the communicator is the calling process? You suggest that an application should call MPI_Comm_check/validate before calling the MPI_Recv(ANY)? What if a failure occurs between these two operations that would have changed the decision of the application to post the MPI_Recv(ANY) - e.g., the caller is the only process left in the communicator? There remains problems with threading, but much of those problems are derived from the above questions. (6) ok (7) I do not really follow why MPI_Comm_check is better than _validate? Can you elaborate? How does an Allreduce provide you the same semantic guarantees as a MPI_Comm_check/validate? A process might have failed in the tree during the result distribution phase causing dependent processes to fail the Allreduce operation. The point is that MPI_Allreduce is not a fault tolerant agreement algorithm, MPI_Comm_check/validate is. They may have the same failure free complexity, but they are semantically different. (8) The reason we want to allow for the option of reusing a communicator with holes in the membership is for application indexing. This was something that applications requested as many of them use the rank in the communicator to calculate offsets into data structures or participation in the operation. So keeping their rank was seen as important. An application that wants to can create a new communicator with only the alive ranks, but is not required to do so. A different tact, MPI_COMM_WORLD will have holes in it upon process failure, so is it valid to call a collective operation (like MPI_Comm_create) over it to get the new shrunken communicator? (9) Applications may want to keep track of failed processes for application indexing, as just mentioned, but also for later recovery (you need to know who failed if you are going to recover them). (10) I don't think the semantics for MPI_Comm_size are odd at all. MPI_Comm_size returns the total size of the communicator, a value that is often used to setup data structures of appropriate size and to reason about the extent to names in the communicator. So, since communicators have a static size, MPI_Comm_size should always return that static size. If you only track the alive processes, how do you know from which you should recover? (11) So communicator creation should still be 'transactional' - meaning that it returns uniformly at all processes? On Fri, Jan 13, 2012 at 4:41 PM, Sur, Sayantan <sayantan.sur(a)intel.com>wrote: > FT WG,**** > > ** ** > > I would like to thank Josh for enduring the marathon plenary presentation! > It was truly commendable.**** > > ** ** > > Based on the Forum feedback and vote, it is apparent that there are some > significant issues. Primarily due to several new concepts and terms, that > the larger Forum does not believe to be required, OR present implementation > challenges for the rest of MPI library.**** > > ** ** > > I would like to argue for a simplified version of the proposal that covers > a large percentage of use-cases and resists adding new “features” for the > full-range of ABFT techniques. It is good if we have a more pragmatic view > and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps > just want to do something like this:**** > > ** ** > > for(… really long time …) {**** > > MPI_Comm_check(work_comm, &is_ok, &alive_group);**** > > if(!is_ok) {**** > > MPI_Comm_create_group(alive_group, …, &new_comm);**** > > // re-balance workload and use new_comm in rest of computation**** > > MPI_Comm_free(work_comm); // get rid of old comm**** > > work_comm = new_comm;**** > > } else {**** > > // continue computation using work_comm**** > > // if some proc failed in this iteration, roll back work done in this > iteration, go back to loop**** > > }**** > > }**** > > ** ** > > Here are some modifications I would like to propose to the current chapter > (in order as these concepts/terms appear in the text):**** > > ** ** > > **1. **Remove concept of “recognized failed” processes. As was > pointed out in the meeting, we don’t really care about the failed > processes, rather the alive ones. Accordingly, rename > MPI_Comm(win/file)_validate() to MPI_Comm(win/file)_check(MPI_Comm comm, > int * is_ok, MPI_Group * alive_group);**** > > **2. **Remove concept of “collectively inactive/active”. This > doesn’t really bring anything to the table, rather conflicts with existing > definition of collectives. MPI defines collectives as being equivalent of a > series of point-to-point calls. As per that definition, if the > point-to-point calls succeed (i.e. the corresponding processes are alive), > then as locally observed, collective call has also succeeded. As far as the > application is concerned as long as the local part of collective is > complete successfully, it is OK. If they want to figure out global status, > they can always call MPI_Comm_check() or friends.**** > > **3. **Eventually perfect failure detector/strongly > complete/strongly accurate/etc: We replace this discussion (even remove > much of 17.3) with a much more straight-forward requirement – > “Communication with a process completes with either success or error. In > case of communication with failed processes, communication calls and > requests may complete with MPI_ERR_PROC_FAILSTOP.” Note that MPI standard > requires all communication to complete before calling MPI_Finalize – > therefore, the first part of this requirement is nothing new. The second > part indicates that there is no guarantee that communication with a failed > process **will** fail. Messages may have been internally buffered before > the real failure may still be delivered per existing MPI semantics.**** > > **a. **This does raise the question from implementers: “When do I > mark requests as MPI_ERR_PROC_FAILSTOP? How long do I wait?” The answer > completely depends on the implementation. Obviously, there is some > requirement to deal with process launcher runtime. In some implementations > with connected mode may be able to leverage hw or os techniques to detect > connections that have gone down. MPI implementations using connection-less > transports may need additional work. However, **none** of this is new > work/concepts. As far as possible, we should talk minimally about what the > MPI implementation might do to achieve this.**** > > **4. **Remove process failure handlers – 17.5.1, 17.5.2, 17.5.3, > 17.5.4. The only way to find out if something failed is to call > MPI_Comm_check() and friends. This removes a whole lot of complexity with > failure handlers. Fail handlers can be emulated over this interface as a > library. We may consider them for MPI-3.1 (or 4).**** > > **5. **Point-to-point communication: Remove the concept of > MPI_ERR_ANY_SOURCE_DISABLED and corresponding calls to re-enable any > source. The concept of disabling ANY_SOURCE is counter-intuitive. When an > app/lib posts a recv with ANY_SOURCE, it is specifically telling the MPI > library that **any** source is OK and implicitly means that if some > senders are unable to send, application/lib does not care! Master/slave > type of applications wishing to use FT features can periodically call > MPI_Comm_check(). Additionally, if the master tries to send to the dead > process, it may get an error. My guess is that master/slave type of apps > are among the most resilient, and some even work with the current standard > (MPI_ERRORS_RETURN). A benefit of removing this restriction is that we no > longer have the threading complexities of re-enabling any source using > reader/writer locks J Therefore, we can remove 17.6.3.**** > > **6. **Retain MPI_Comm_drain() and MPI_Comm_idrain() as they > provide useful functionality.**** > > **7. **Collective communication: Rename comm_validate() to > comm_check() as per discussion above. We can keep comm_check_multiple() as > it provides useful functionality for overlapping communicators by reducing > overhead to check them. We can retain much of 17.7.2 while removing > references to “collectively inactive”. If the output of collective depends > on contribution from a failed process, then obviously, the collective > fails. This is in keeping with point-to-point semantics – one cannot > receive any data from a failed process. Keep in mind if the contribution > from failed process may have arrived before it failed – and that is OK (not > flagged as failure). Some collectives, such as MPI_Bcast, may succeed even > if processes down the bcast tree have failed as sends may simply be > buffered. The app/lib will only know if a collective was a global success > by either performing an Allreduce after the collective OR calling > comm_check(). In any case, it is left to app/lib and not MPI to report > failures of processes the library didn’t try to communicate with during > this op.**** > > **8. **I am proposing that once a collective fails with > MPI_ERR_PROC_FAIL_STOP, all subsequent collectives on that comm fail > immediately with MPI_ERR_PROC_FAIL_STOP. App/lib needs to use > MPI_Comm_create_group() to fork off a new comm of live procs and continue > with it. This is a deviation from the current proposal that allows > collectives on bad comms (after re-enabling collectives) and keeps 0s as > contributions. I am aware that this might not fully satisfy all use cases > (although at this point of time, I cannot think of any), but in a broader > view, we could think this of as a compromise to reduce complexity.**** > > **9. **Example 17.4 changes only slightly to call comm_check() and > then split off the new communicator. Why keep failed procs in the > communicator anyways?**** > > **10. **Note that this change in semantics allows us to bypass the > question raised: “Why does comm_size() on a communicator with failed procs > still return the old value- alive_ranks + failed_ranks?” As I mentioned > before, this is odd, and we should encourage app/lib to only deal with > known alive ranks. The current proposal does the reverse – forces app to > keep track of “known failed”. This causes confusion!**** > > **11. **Process topologies 17.9 – should change to say that we can only > use communicators with live ranks. i.e. if you know your comm was bad, > split off a new comm with live ranks. During the op, some ranks may fail – > and that is OK since MPI_ERR_PROC_FAIL_STOP will be raised. This is > mentioned in the current proposal.**** > > **12. **Similar changes in semantics to windows and files.**** > > ** ** > > Please let me know if I overlooked some corner cases or I have > mis-interpreted the text of the current chapter. I gave it some thought, > but WG knows best!**** > > ** ** > > ** ** > > Thanks!**** > > ** ** > > ===**** > > Sayantan Sur, Ph.D.**** > > Intel Corp.**** > > ** ** > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] simplified FT proposal
by Josh Hursey 17 Jan '12

17 Jan '12

Bill, (A bit of an aside) I completely agree that creating a fault tolerant application is a tricky endeavor even for the most heroic of developers. Developing a 'self-stabilizing' application is difficult, and will require extensive experimentation to derive appropriate algorithms, applications, and libraries. Some work has happened in this space, but there is a great need for more research. I worry about specifying something that is too restrictive to developer (overreaching our responsibilities in a sense), and thus stifling what researchers can experiment with. The way I have approached the task of defining a fault tolerant MPI standard is by asking 'how should this specific interface behave when a process fails?' When there are multiple options then I have tried to gather application preference and further comment. Then try to weave those specific solutions into a pattern that can be applied throughout the standard for consistency. I believe that the majority of the RTS proposal is correct in this regard, and most of the contention seems to be over specific choices when multiple solutions are on the table (ANY_SOURCE is a great example). To that end we have to better articulate the decision process. There are other features that seem reaching, and we need to assess if those are necessary or supplementary. You are correct that the information you receive from a MPI_Comm_check/validate call is only representative of the known failures at the time of the call. So it is likely that additional processes failed just as the validate operation finishes making the data old. The RTS proposal modified/clarified the semantics of MPI_Comm_split() so that if the communicator is created then it contains only the alive processes that called the operation. Of course additional processes may have failed just after creation, but there is nothing we can do about that. Is such an operation what you are looking for? -- Josh On Sun, Jan 15, 2012 at 5:40 PM, William Gropp <wgropp(a)illinois.edu> wrote: > One concern that I have with fault tolerant proposals has to do with races > in the specification. This is an area where users often "just want it to > work" but getting it right is tricky. In the example here, the > "alive_group" is really only that at some moment shortly before > "MPI_Comm_check" returns (and possibly not even that). After that, it is > really the "group_of_processes_that_was_alive_at_some_point_in_the_past". > Since there are sometimes correlations in failures, this could happen even > if the initial failure is rare. An alternate form might be to have a > routine, collective over a communicator, that returns a new communicator > meeting some definition of "members were alive at some point during > construction". It wouldn't guarantee you could use it, but it would have > cleaner semantics. > > Bill > > On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote: > > I would like to argue for a simplified version of the proposal that covers > a large percentage of use-cases and resists adding new “features” for the > full-range of ABFT techniques. It is good if we have a more pragmatic view > and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps > just want to do something like this:**** > ** ** > for(… really long time …) {**** > MPI_Comm_check(work_comm, &is_ok, &alive_group);**** > if(!is_ok) {**** > MPI_Comm_create_group(alive_group, …, &new_comm);**** > // re-balance workload and use new_comm in rest of computation**** > MPI_Comm_free(work_comm); // get rid of old comm**** > work_comm = new_comm;**** > } else {**** > // continue computation using work_comm**** > // if some proc failed in this iteration, roll back work done in this > iteration, go back to loop**** > }**** > }**** > ** ** > > > William Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization
by Rolf Rabenseifner 17 Jan '12

17 Jan '12

Dear committees of - FT, - MPI_Init --> 8. Environmental Management, - MPI_Init_thread --> 12. External Interfaces. Before discussing details, I would like to get a clear answer whether you believe that the proposal below is a good or bad idea. As already mentioned at the Jan. 2012 meeting, I would like to propose that the FT group may substitute the unclear collective behavior of MPI_Failhandler_set_mode by adding the mode to the MPI initialization. For this, I added a proposal to slide 4 in MPI_Forum_Overview_MPI-3.0_Jan2012_action-items.ppt (see my previous mail to the MPI-Forum list) - If appropriate, a new ticket that enhances MPI_INIT_THREAD -- Required and provided as “bit vector” of "bit-wise OR" of required_/provided_threadsafety | required_/provided_ft_mode | required_/provided_cancel_mode | required_/provided_any_source_mode -- New mask-constants MPI_THREAD_MASK, MPI_FT_MASK, MPI_CANCEL_MASK, MPI_ANY_SOURCE_MASK -- With existing values for required_/provided_threadsafety MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, MPI_THREAD_SERIALIZED, and MPI_THREAD_MULTIPLE. -- With new values for - required_/provided_ft_mode = MPI_FT_NONE=0, or MPI_FT_FAILHANDLER_MODE_ALL≠0, or MPI_FT_FAILHANDLER_MODE_SUBSET≠0, or - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED=0, or MPI_NO_CANCEL≠0 - required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED=0, or MPI_NO_ANY_SOURCE≠0 -- Values must be set identical for all processes in an MPI_COMM_WORLD - It is easier to relax about this in further versions of MPI than to relax already now and to restrict later as now done in ticket #222 for required_/provided_threadsafety - For each of the for "variables", a different decision can be done. -- At least for required_/provided_cancel_mode and ...any_source_mode, I would require that the provided value must be identical to the required value. Reason: Internally, the value can be ignored. -- For required_/provided_ft_mode, I would recommend to allow that provided_ft_mode must be - identical to required_ft_mode or MPI_FT_NONE - and the same in all processes. -- MPI_INIT_THREAD & MPI_QUERY_THREAD moves - from 12.4.3 (External Interfaces) - to 8.7 (Startup) - but explanations to ...threadsafety | ...ft_mode | ...cancel_mode | ...any_source_mode are kept or written in the appropriate sections 12.4.3, new 17.5 (FT Environm.), 3.8.4 (Cancel), 3.2.4 (Blocking Receive) -- A call to MPI_INIT is identical to MPI_INIT_THREAD with - the rules in 12.4.3 about required_/provided_threadsafety - required_/provided_ft_mode = MPI_FT_NONE, - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED, - required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED -- This ticket would have the following properties: - It is clearly source-code and ABI backward compatible, because - the values of MPI_THREAD_SINGLE, _FUNNELED, ... need not to be changed and MPI_THREAD_SINGLE need not to be zero; - the values representing the current MPI-2.2 quality are set to zero: MPI_FT_NONE=0, MPI_CANCEL_ALLOWED=0, and MPI_ANY_SOURCE_ALLOWED=0 - It is very unlikely that an implementation has used more than 24 different bits in these 4 integer constants MPI_THREAD_SINGLE, ... _MULTIPLE. Therefore MPI_THREAD_MASK would have a maximum of 24 bits. Enough room for the 4 bits needed together for the other three ..._MASKs. - FT can be switched on or off at the MPI initialization and is switched off in unchanged applications. Therefore no backward-compatibility-problem with a modified behavior of the default error handlers when FT is switched on. - Normally there should be enough bit-space for further decisions at MPI initialization. - The decisions about the cancel and any_source values would be done in different tickets - FT quality is optional if we add the rule that provided_ft_mode may be identical to required_ft_mode ***or*** MPI_FT_NONE - This rule can be changed in a further version of MPI without backward-compatibility-problems. I would like to get a reply from - the FT group - the chapter committee of MPI_Init --> 8. Environmental Manag. George Bosilca(c), Josh Hursey, Terry Dontje - the chapter committee of MPI_Init_thread --> 12. External Interf. Bronis R. de Supinski(c), Pavan Balaji Before discussing details, I would like to get a clear answer whether you believe that this is a good or bad idea. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [Mpi3-ft] simplified FT proposal
by Graham, Richard L. 16 Jan '12

16 Jan '12

Before commenting in detail, what has been proposed is the result of quite a bit of work to figure out what apps want or need over about a 4 year period (with about 1 year hiatus), so quite a bit of work has been done to get input from various groups, and the resulting proposal reflects this. As a standard, MPI must satisfy the broad set of users, and not select a subset to support. There are two classes of error notification approaches: - notify only those directly affected by the the failure - notify a broader set of process - user specified. This is the failure handler concept added recently, that has been demonstrated to be useful in the context of linear FT linear algebra. For the first cast the principals are: - only those directly affected by the errors are notified, via a return error code. For collective communication, this implies that anyone using collective operations is notified after failure occurs. When wild card receives are posted, the library can't guess if there is a problem or not, so as long as there is no communication to process, some sort of notification needs to occur, to avoid deadlock. - Collective communications are expensive, so, under the assumption that failures are rare relative to the number of collective calls, validate is used to amortize the cost of collective integrity checks over several iterations, guided by the user. There has ALWAYS been the intent to let user ask to use more robust collectives, but this just did not make it into this version of the proposal. So, there are actually a very small number of principals, but a large standard, so need to deal with the implications throughout the entire standard ... More specific comments in line. On Jan 13, 2012, at 4:41 PM, Sur, Sayantan wrote: FT WG, I would like to thank Josh for enduring the marathon plenary presentation! It was truly commendable. Based on the Forum feedback and vote, it is apparent that there are some significant issues. Primarily due to several new concepts and terms, that the larger Forum does not believe to be required, OR present implementation challenges for the rest of MPI library. I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and resists adding new ï¿½featuresï¿½ for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this: for(ï¿½ really long time ï¿½) { MPI_Comm_check(work_comm, &is_ok, &alive_group); if(!is_ok) { MPI_Comm_create_group(alive_group, ï¿½, &new_comm); // re-balance workload and use new_comm in rest of computation MPI_Comm_free(work_comm); // get rid of old comm work_comm = new_comm; } else { // continue computation using work_comm // if some proc failed in this iteration, roll back work done in this iteration, go back to loop } } [rich] Actually, this is only one use scenario, and one we deliberately did not want to require folks use. There is a specific goal to allow completely local recovery for those apps that do not need global consistency. This was one of the very early requirements. This is where I repeat that MPI needs to support a range of application scenarios, as there is not a single application style in use. Here are some modifications I would like to propose to the current chapter (in order as these concepts/terms appear in the text): 1. Remove concept of ï¿½recognized failedï¿½ processes. As was pointed out in the meeting, we donï¿½t really care about the failed processes, rather the alive ones. Accordingly, rename MPI_Comm(win/file)_validate() to MPI_Comm(win/file)_check(MPI_Comm comm, int * is_ok, MPI_Group * alive_group); [rich] It probably depends on the app, as to whether or not it cares about the failed processes. In a client/server scenario, one cares a lot if the failure is in the server, or one of the clients. 2. Remove concept of ï¿½collectively inactive/activeï¿½. This doesnï¿½t really bring anything to the table, rather conflicts with existing definition of collectives. MPI defines collectives as being equivalent of a series of point-to-point calls. As per that definition, if the point-to-point calls succeed (i.e. the corresponding processes are alive), then as locally observed, collective call has also succeeded. As far as the application is concerned as long as the local part of collective is complete successfully, it is OK. If they want to figure out global status, they can always call MPI_Comm_check() or friends. [rich] your point is correct - MPI currently defines the collective over the entire set of ranks, so to proceed with a smaller set of ranks, the apps need to be involved - only they know if this has any value. The terms "collectively inactive/active" are a convenient short hand to express whether or not collective communication can be used - rather than going through a longer explanation every time this concept is mentioned in the text. I suppose that your assertion is that collectives are useless if not all ranks participate - however this is contrary to how some apps use collectives. There is no one-size fits all here. 3. Eventually perfect failure detector/strongly complete/strongly accurate/etc: We replace this discussion (even remove much of 17.3) with a much more straight-forward requirement ï¿½ ï¿½Communication with a process completes with either success or error. [rich] So, Josh actually did quite a bit of work to ground what we are doing in the broader context of FT. This actually conveys very specific requirements on the implementation - error notification must occur ... In case of communication with failed processes, communication calls and requests may complete with MPI_ERR_PROC_FAILSTOP.ï¿½ Note that MPI standard requires all communication to complete before calling MPI_Finalize ï¿½ therefore, the first part of this requirement is nothing new. The second part indicates that there is no guarantee that communication with a failed process *will* fail. Messages may have been internally buffered before the real failure may still be delivered per existing MPI semantics. [rich] MPI completion semantics are not changed at all with this. a. This does raise the question from implementers: ï¿½When do I mark requests as MPI_ERR_PROC_FAILSTOP? How long do I wait?ï¿½ The answer completely depends on the implementation. Obviously, there is some requirement to deal with process launcher runtime. In some implementations with connected mode may be able to leverage hw or os techniques to detect connections that have gone down. MPI implementations using connection-less transports may need additional work. However, *none* of this is new work/concepts. As far as possible, we should talk minimally about what the MPI implementation might do to achieve this. [rich] I do not believe any implementation details are specified. What is specified is that error notification must eventually occur. The implication is that this is in a finite period of time, but does not specify what this period is. 4. Remove process failure handlers ï¿½ 17.5.1, 17.5.2, 17.5.3, 17.5.4. The only way to find out if something failed is to call MPI_Comm_check() and friends. This removes a whole lot of complexity with failure handlers. Fail handlers can be emulated over this interface as a library. We may consider them for MPI-3.1 (or 4). [rich] Why remove support of a class of applications that has already used this approach ? This was added very recently to support the broader range of application scenarios, rather than focus on the first type of error notification. 5. Point-to-point communication: Remove the concept of MPI_ERR_ANY_SOURCE_DISABLED and corresponding calls to re-enable any source. The concept of disabling ANY_SOURCE is counter-intuitive. When an app/lib posts a recv with ANY_SOURCE, it is specifically telling the MPI library that *any* source is OK and implicitly means that if some senders are unable to send, application/lib does not care! [rich] this is actually wrong. posting an any-source request says that data from any source may be matched, but says nothing at all about what to do if failure occurs. I have direct experience with apps written on the order of ten years ago, where all data was received as any-source, so a process failure with no means of notification on failure would cause such apps to hang. We did feel that if an error occurred after an any-source was posted, that the user should have input as to whether or not to continue. Master/slave type of applications wishing to use FT features can periodically call MPI_Comm_check(). Additionally, if the master tries to send to the dead process, it may get an error. My guess is that master/slave type of apps are among the most resilient, and some even work with the current standard (MPI_ERRORS_RETURN). A benefit of removing this restriction is that we no longer have the threading complexities of re-enabling any source using reader/writer locks :) Therefore, we can remove 17.6.3. 6. Retain MPI_Comm_drain() and MPI_Comm_idrain() as they provide useful functionality. 7. Collective communication: Rename comm_validate() to comm_check() as per discussion above. We can keep comm_check_multiple() as it provides useful functionality for overlapping communicators by reducing overhead to check them. We can retain much of 17.7.2 while removing references to ï¿½collectively inactiveï¿½. If the output of collective depends on contribution from a failed process, then obviously, the collective fails. This is in keeping with point-to-point semantics ï¿½ one cannot receive any data from a failed process. Keep in mind if the contribution from failed process may have arrived before it failed ï¿½ and that is OK (not flagged as failure). Some collectives, such as MPI_Bcast, may succeed even if processes down the bcast tree have failed as sends may simply be buffered. The app/lib will only know if a collective was a global success by either performing an Allreduce after the collective OR calling comm_check(). In any case, it is left to app/lib and not MPI to report failures of processes the library didnï¿½t try to communicate with during this op. [rich] The library generally has much better access to error information, so it makes sense to get help from the library on this. Many apps expect collectives to complete over the entire communicator, so if this is not going to be the case, apps need to opt in to use these. 8. I am proposing that once a collective fails with MPI_ERR_PROC_FAIL_STOP, all subsequent collectives on that comm fail immediately with MPI_ERR_PROC_FAIL_STOP. App/lib needs to use MPI_Comm_create_group() to fork off a new comm of live procs and continue with it. This is a deviation from the current proposal that allows collectives on bad comms (after re-enabling collectives) and keeps 0s as contributions. I am aware that this might not fully satisfy all use cases (although at this point of time, I cannot think of any), but in a broader view, we could think this of as a compromise to reduce complexity. [rich] nothing prevents apps from doing this. But there are apps that don't need this, so why add extra work for them ? 9. Example 17.4 changes only slightly to call comm_check() and then split off the new communicator. Why keep failed procs in the communicator anyways? 10. Note that this change in semantics allows us to bypass the question raised: ï¿½Why does comm_size() on a communicator with failed procs still return the old value- alive_ranks + failed_ranks?ï¿½ As I mentioned before, this is odd, and we should encourage app/lib to only deal with known alive ranks. The current proposal does the reverse ï¿½ forces app to keep track of ï¿½known failedï¿½. This causes confusion! [rich] The choice to keep mpi_comm_size/rank() returning the same value, regardless of the state of the communicator is to avoid forcing ranks unaffected by the failure to have to do something when failure happens. Rich 11. Process topologies 17.9 ï¿½ should change to say that we can only use communicators with live ranks. i.e. if you know your comm was bad, split off a new comm with live ranks. During the op, some ranks may fail ï¿½ and that is OK since MPI_ERR_PROC_FAIL_STOP will be raised. This is mentioned in the current proposal. 12. Similar changes in semantics to windows and files. Please let me know if I overlooked some corner cases or I have mis-interpreted the text of the current chapter. I gave it some thought, but WG knows best! Thanks! === Sayantan Sur, Ph.D. Intel Corp. _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] simplified FT proposal
by Anthony Skjellum 16 Jan '12

16 Jan '12

More #5) A parallel thread, running on the same group as the communicator used for the user program in FTBLOCK, would be free to be running a gossip protocol, instantiated a) by the user at his/her choice b) by MPI at its choice, by user demand This parallel inspector thread propagates error state, and accepts input to the error state by allowing user programs to assert error state within the FTBLOCK. You can think of this parallel thread as cooperation of the progress engines with gossip, or as a completely optional activity, created by the user, or as a part of the FT proposal. As long as there are local ways to add error state, we can write this as a layered library for the sake of making sure that local errors are propagated (remembering that MPI is returning local errors to the user inside his/her FTBLOCK). If we want the implementation to provide/support this kind of FT-friendly error propagation, that is a bigger step, which I am not advocating as required. I think it would be nice to have this be part of a fault tolerant communication space. But, I am OK with users doing this for themselves, esp. because they can control granularity better. #6) I think MPI_COMM_WORLD is a bummer in the FT world. We don't want it to hang around very long after we get going. If we are really working on subgroups, and these subgroups form a hierarchical graph, rather than an all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD after the first process breaks. So - as an example of pain of MPI scalability - the build-down model of MPI-1 is less good for this than a build-up model, where we find a way for groups to rendezvous. Obviously, for convenience, the all-to-all virtual topology in the onset of MPI_Init() is nice, but I am assuming that errors may happen quite quickly. MPI_Init() with no failures is only good for a limited window of time, given failure rates. During this time, we would like to "turn off MPI_COMM_WORLD" unless we really don't need it, or at least not have to reconstruct it ever if we don't need it. So, we need to agree on a fault-friendly MPI_Init()... An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of spawning MPI, or the spawn command that creates the world, create an effective communicator. We should generalize MPI_Init() to support outcomes other than pure success, such as partial success (e.g., smaller world than stipulated). A really good approach would be to only create the actual communicators you really need to run your process, not the virtual-all-to-all world of MPI-1 and MPI-2. But, that is a heresy, so forgive me. Tony On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum <tony(a)runtimecomputing.com > wrote: > Satantan, > > Here are my further thoughts, which hopefully includes an answer to your > questions :-) > > 1) Everything is set to "ERRORS return"; the goal is to get local errors > back if they are available. > > 2) I would emphasize non-blocking operations, but blocking operations > implemented with internal timeout could return a timeout-type error > > 3) You don't have to return the same return code, or the same results in > all processes in the communicator, you can get erroneous results > or local failures; the functions are also allowed to produce incorrect > results [and we should then discuss what error reporting means here... > I am happy with local errors returned where known, recognizing those > processes may die before the bottom of the block. However, I also > expect the implementation to do its best to propagate error state > knowledge within this FTBLOCK, based organically on ongoing communication > or on gossip if an implementation so chose.] > > Also, because we assume that there is also algorithmic fault tolerance > at work, local errors may be raised by the application because it is doing > checks > for validity etc. > > So, either MPI or the application may raise local errors prior to the > bottom of the FTBLOCK, and the success of the bottom of the block must be > allowed > to be fail, just based on ABFT inputs from the application to MPI, not > just based on MPI's opinion. > > 4) If you are willing to do everything nonblocking, then I can describe > the test at the bottom of the FTBLOCK as follows: > > The test operation at the bottom of the FTBLOCK is effectively a > generalized WAIT_ALL, that completes or doesn't complete all the outstanding > requests, returning errors related to the faults observed, and providing a > unified 0/1 success failure state consistently across the group of comm > [or surviving members thereof]. > > In my view, the application as well as MPI can contribute error state as > input to the FTBLOCK test. > > Also, the application that gets local errors inside the loop already is > immediately ready to punt at that point, and do a jump to the bottom of the > loop. Let's > assume it is required to do all the operations before getting to the > bottom of the loop, for now, and we just allow that some of these may > return further > errors (I am trying to keep it simple), and MPI-like rules of all > attempting the operations. If we can get this to work, we can weaken it > later. > > There is no good way to describe some BLOCKING, some nonblocking > operations, because we have no descriptor to tell us if something failed > that previously > returned... and did not give a local error, so I am not going to pursue > BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this > later, > if we can get a consistent solution using all nonblocking. > > Please tell me what you think. > > Thanks for responding! > > Tony > > > On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan <sayantan.sur(a)intel.com>wrote: > >> Hi Tony,**** >> >> ** ** >> >> In the example semantics you mentioned, are the “ops” required to return >> the same result on all processors? Although this doesn’t change the API >> “op”, but it does change completion semantics of almost all MPI ops. I hope >> I am correctly interpreting your message.**** >> >> ** ** >> >> Thanks.**** >> >> ** ** >> >> ===**** >> >> Sayantan Sur, Ph.D.**** >> >> Intel Corp.**** >> >> ** ** >> >> *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: >> mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *Anthony Skjellum >> *Sent:* Sunday, January 15, 2012 7:06 PM >> >> *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> *Subject:* Re: [Mpi3-ft] simplified FT proposal**** >> >> ** ** >> >> Everyone, I think we need to start from scratch.**** >> >> ** ** >> >> We should look for minimal fault-tolerant models that are achievable and >> useful. They may allow for a combination of faults (process and network), >> but in the end, as discussed in San Jose:**** >> >> ** ** >> >> FTBLOCK**** >> >> --------------**** >> >> Start_Block(comm)**** >> >> ** ** >> >> op [normal MPI operation on communicator specified by Start_Block, or >> subset thereof]**** >> >> op**** >> >> op**** >> >> ** ** >> >> Test_Block(comm)**** >> >> ** ** >> >> Which either succeeds, or fails, on the whole list of operations, >> followed by ways to reconstruct communicators and add back processes >> (easily),**** >> >> provides for 3-level fault tolerant model**** >> >> ** ** >> >> a) Simply retry if the kind of error at the Test_Block is retryable**** >> >> b) Simply reconstruct the communicator, use algorithmic fault tolerance >> to get lost data, and retry the block**** >> >> c) drop back to 1 or more levels of checkpoint-restart.**** >> >> ** ** >> >> We can envision in this model an unrolling of work, in terms of a >> parameter N, if there is a lot of vector work, to allow granularity control >> **** >> >> as a function of fault environment.**** >> >> ** ** >> >> In some sense, a simpler model, that provides for efforts to detect**** >> >> a) by MPI**** >> >> b) allowing application monitor to assert failure asynchronously to this >> loop**** >> >> ** ** >> >> provides a more general ability to have coverage of faults, including but >> not limited to process faults and possible network faults.**** >> >> ** ** >> >> It changes the API very little.**** >> >> ** ** >> >> It motivates the use of buffers, not zero copy, to support the fact that >> you may to roll-back a series of operations, thereby revealing fault-free >> overhead directly.**** >> >> ** ** >> >> Start_Block and Test_Block are collective and synchronizing, such as >> barriers. Because we have uncertainty to within a message, multiple >> barriers (as mentioned by George Bosilca K to me in a sidebar at the >> meeting).**** >> >> ** ** >> >> We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we >> can. Once we have this stable intermediate form, we explore more**** >> >> options.**** >> >> ** ** >> >> I think it is important to recognize that the reconstruction step, >> including readding processes and making new communicators may mean smarter >> Join operations. It is clear we need to be able to treat failures during >> the recovery process, and use a second level loop, possibly bombing out** >> ** >> >> to checkpoint, if we cannot make net progress on recovery because of >> unmodeled error issues.**** >> >> ** ** >> >> The testing part leverages all the learning so far, but needn't be >> restricted to modeled errors like process faults. There can be modeled and >> unmodeled faults. Based on what fault comes up, the user application then >> has to decide how hard a retry to do, whether just to add processes,**** >> >> whether just to retry the loop, whether to go to a checkpoint, whether to >> restart the APP. MPI could give advice, based on its understanding of*** >> * >> >> the fault model, in terms of sufficient conditions for "working harder" >> vs "trying the easiest" for fault models it understands somewhat for a >> system.**** >> >> ** ** >> >> Now, the comments here are a synopsis of part of the side bars and open >> discussion we had in San Jose, distilled a bit. I want to know why **** >> >> we can't start with this, succeed with this, implement and test it, and >> having succeeded, do more in a future 3.y, y > x release, given user >> experience.**** >> >> ** ** >> >> I am not speaking to the choice of "killing all communicators" as with >> FT-MPI, or "just remaking those you need to remake." I think we need to >> resolve. Honestly, groups own the fault property, not communicators, and >> all groups held by communicators where the fault happened should be >> rebuilt, not all communicators... Let's argue on that.**** >> >> ** ** >> >> So, my suggestion is REBOOT the proposal with something along lines >> above, unless you all see this is no better.**** >> >> ** ** >> >> With kind regards,**** >> >> Tony**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan <sayantan.sur(a)intel.com> >> wrote:**** >> >> Hi Bill,**** >> >> **** >> >> I am in agreement with your suggestion to have a collective over a >> communicator that returns a new communicator containing ranks “alive some >> point during construction”. It provides cleaner semantics. The example was >> merely trying to utilize the new MPI_Comm_create_group API that the Forum >> is considering.**** >> >> **** >> >> MPI_Comm_check provides a method to form global consensus in that all >> ranks in comm did call it. It does not imply anything about current status >> of comm, or even the status “just before” the call returns. During the >> interval before the next call to MPI_Comm_check, it is possible that many >> ranks fail. However, the app/lib using MPI knows the point where everyone >> was alive.**** >> >> **** >> >> Thanks.**** >> >> **** >> >> ===**** >> >> Sayantan Sur, Ph.D.**** >> >> Intel Corp.**** >> >> **** >> >> *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: >> mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *William Gropp >> *Sent:* Sunday, January 15, 2012 2:41 PM >> *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> *Subject:* Re: [Mpi3-ft] simplified FT proposal**** >> >> **** >> >> One concern that I have with fault tolerant proposals has to do with >> races in the specification. This is an area where users often "just want >> it to work" but getting it right is tricky. In the example here, the >> "alive_group" is really only that at some moment shortly before >> "MPI_Comm_check" returns (and possibly not even that). After that, it is >> really the "group_of_processes_that_was_alive_at_some_point_in_the_past". >> Since there are sometimes correlations in failures, this could happen even >> if the initial failure is rare. An alternate form might be to have a >> routine, collective over a communicator, that returns a new communicator >> meeting some definition of "members were alive at some point during >> construction". It wouldn't guarantee you could use it, but it would have >> cleaner semantics.**** >> >> **** >> >> Bill**** >> >> **** >> >> On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote:**** >> >> ** ** >> >> I would like to argue for a simplified version of the proposal that >> covers a large percentage of use-cases and resists adding new “features” >> for the full-range of ABFT techniques. It is good if we have a more >> pragmatic view and not sacrifice the entire FT proposal for the 1% fringe >> cases. Most apps just want to do something like this:**** >> >> **** >> >> for(… really long time …) {**** >> >> MPI_Comm_check(work_comm, &is_ok, &alive_group);**** >> >> if(!is_ok) {**** >> >> MPI_Comm_create_group(alive_group, …, &new_comm);**** >> >> // re-balance workload and use new_comm in rest of computation**** >> >> MPI_Comm_free(work_comm); // get rid of old comm**** >> >> work_comm = new_comm;**** >> >> } else {**** >> >> // continue computation using work_comm**** >> >> // if some proc failed in this iteration, roll back work done in >> this iteration, go back to loop**** >> >> }**** >> >> }**** >> >> **** >> >> **** >> >> William Gropp**** >> >> Director, Parallel Computing Institute**** >> >> Deputy Director for Research**** >> >> Institute for Advanced Computing Applications and Technologies**** >> >> Paul and Cynthia Saylor Professor of Computer Science**** >> >> University of Illinois Urbana-Champaign**** >> >> **** >> >> **** >> >> **** >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft**** >> >> >> >> **** >> >> ** ** >> >> -- >> Tony Skjellum, PhD >> RunTime Computing Solutions, LLC >> tony(a)runtimecomputing.com >> direct: +1-205-314-3595 >> cell: +1-205-807-4968**** >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968 > > -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968

1 0

Re: [Mpi3-ft] simplified FT proposal
by Anthony Skjellum 16 Jan '12

16 Jan '12

Satantan, Here are my further thoughts, which hopefully includes an answer to your questions :-) 1) Everything is set to "ERRORS return"; the goal is to get local errors back if they are available. 2) I would emphasize non-blocking operations, but blocking operations implemented with internal timeout could return a timeout-type error 3) You don't have to return the same return code, or the same results in all processes in the communicator, you can get erroneous results or local failures; the functions are also allowed to produce incorrect results [and we should then discuss what error reporting means here... I am happy with local errors returned where known, recognizing those processes may die before the bottom of the block. However, I also expect the implementation to do its best to propagate error state knowledge within this FTBLOCK, based organically on ongoing communication or on gossip if an implementation so chose.] Also, because we assume that there is also algorithmic fault tolerance at work, local errors may be raised by the application because it is doing checks for validity etc. So, either MPI or the application may raise local errors prior to the bottom of the FTBLOCK, and the success of the bottom of the block must be allowed to be fail, just based on ABFT inputs from the application to MPI, not just based on MPI's opinion. 4) If you are willing to do everything nonblocking, then I can describe the test at the bottom of the FTBLOCK as follows: The test operation at the bottom of the FTBLOCK is effectively a generalized WAIT_ALL, that completes or doesn't complete all the outstanding requests, returning errors related to the faults observed, and providing a unified 0/1 success failure state consistently across the group of comm [or surviving members thereof]. In my view, the application as well as MPI can contribute error state as input to the FTBLOCK test. Also, the application that gets local errors inside the loop already is immediately ready to punt at that point, and do a jump to the bottom of the loop. Let's assume it is required to do all the operations before getting to the bottom of the loop, for now, and we just allow that some of these may return further errors (I am trying to keep it simple), and MPI-like rules of all attempting the operations. If we can get this to work, we can weaken it later. There is no good way to describe some BLOCKING, some nonblocking operations, because we have no descriptor to tell us if something failed that previously returned... and did not give a local error, so I am not going to pursue BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this later, if we can get a consistent solution using all nonblocking. Please tell me what you think. Thanks for responding! Tony On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan <sayantan.sur(a)intel.com>wrote: > Hi Tony,**** > > ** ** > > In the example semantics you mentioned, are the “ops” required to return > the same result on all processors? Although this doesn’t change the API > “op”, but it does change completion semantics of almost all MPI ops. I hope > I am correctly interpreting your message.**** > > ** ** > > Thanks.**** > > ** ** > > ===**** > > Sayantan Sur, Ph.D.**** > > Intel Corp.**** > > ** ** > > *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *Anthony Skjellum > *Sent:* Sunday, January 15, 2012 7:06 PM > > *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > *Subject:* Re: [Mpi3-ft] simplified FT proposal**** > > ** ** > > Everyone, I think we need to start from scratch.**** > > ** ** > > We should look for minimal fault-tolerant models that are achievable and > useful. They may allow for a combination of faults (process and network), > but in the end, as discussed in San Jose:**** > > ** ** > > FTBLOCK**** > > --------------**** > > Start_Block(comm)**** > > ** ** > > op [normal MPI operation on communicator specified by Start_Block, or > subset thereof]**** > > op**** > > op**** > > ** ** > > Test_Block(comm)**** > > ** ** > > Which either succeeds, or fails, on the whole list of operations, followed > by ways to reconstruct communicators and add back processes (easily),**** > > provides for 3-level fault tolerant model**** > > ** ** > > a) Simply retry if the kind of error at the Test_Block is retryable**** > > b) Simply reconstruct the communicator, use algorithmic fault tolerance to > get lost data, and retry the block**** > > c) drop back to 1 or more levels of checkpoint-restart.**** > > ** ** > > We can envision in this model an unrolling of work, in terms of a > parameter N, if there is a lot of vector work, to allow granularity control > **** > > as a function of fault environment.**** > > ** ** > > In some sense, a simpler model, that provides for efforts to detect**** > > a) by MPI**** > > b) allowing application monitor to assert failure asynchronously to this > loop**** > > ** ** > > provides a more general ability to have coverage of faults, including but > not limited to process faults and possible network faults.**** > > ** ** > > It changes the API very little.**** > > ** ** > > It motivates the use of buffers, not zero copy, to support the fact that > you may to roll-back a series of operations, thereby revealing fault-free > overhead directly.**** > > ** ** > > Start_Block and Test_Block are collective and synchronizing, such as > barriers. Because we have uncertainty to within a message, multiple > barriers (as mentioned by George Bosilca K to me in a sidebar at the > meeting).**** > > ** ** > > We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we > can. Once we have this stable intermediate form, we explore more**** > > options.**** > > ** ** > > I think it is important to recognize that the reconstruction step, > including readding processes and making new communicators may mean smarter > Join operations. It is clear we need to be able to treat failures during > the recovery process, and use a second level loop, possibly bombing out*** > * > > to checkpoint, if we cannot make net progress on recovery because of > unmodeled error issues.**** > > ** ** > > The testing part leverages all the learning so far, but needn't be > restricted to modeled errors like process faults. There can be modeled and > unmodeled faults. Based on what fault comes up, the user application then > has to decide how hard a retry to do, whether just to add processes,**** > > whether just to retry the loop, whether to go to a checkpoint, whether to > restart the APP. MPI could give advice, based on its understanding of**** > > the fault model, in terms of sufficient conditions for "working harder" vs > "trying the easiest" for fault models it understands somewhat for a system. > **** > > ** ** > > Now, the comments here are a synopsis of part of the side bars and open > discussion we had in San Jose, distilled a bit. I want to know why **** > > we can't start with this, succeed with this, implement and test it, and > having succeeded, do more in a future 3.y, y > x release, given user > experience.**** > > ** ** > > I am not speaking to the choice of "killing all communicators" as with > FT-MPI, or "just remaking those you need to remake." I think we need to > resolve. Honestly, groups own the fault property, not communicators, and > all groups held by communicators where the fault happened should be > rebuilt, not all communicators... Let's argue on that.**** > > ** ** > > So, my suggestion is REBOOT the proposal with something along lines above, > unless you all see this is no better.**** > > ** ** > > With kind regards,**** > > Tony**** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan <sayantan.sur(a)intel.com> > wrote:**** > > Hi Bill,**** > > **** > > I am in agreement with your suggestion to have a collective over a > communicator that returns a new communicator containing ranks “alive some > point during construction”. It provides cleaner semantics. The example was > merely trying to utilize the new MPI_Comm_create_group API that the Forum > is considering.**** > > **** > > MPI_Comm_check provides a method to form global consensus in that all > ranks in comm did call it. It does not imply anything about current status > of comm, or even the status “just before” the call returns. During the > interval before the next call to MPI_Comm_check, it is possible that many > ranks fail. However, the app/lib using MPI knows the point where everyone > was alive.**** > > **** > > Thanks.**** > > **** > > ===**** > > Sayantan Sur, Ph.D.**** > > Intel Corp.**** > > **** > > *From:* mpi3-ft-bounces(a)lists.mpi-forum.org [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org] *On Behalf Of *William Gropp > *Sent:* Sunday, January 15, 2012 2:41 PM > *To:* MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > *Subject:* Re: [Mpi3-ft] simplified FT proposal**** > > **** > > One concern that I have with fault tolerant proposals has to do with races > in the specification. This is an area where users often "just want it to > work" but getting it right is tricky. In the example here, the > "alive_group" is really only that at some moment shortly before > "MPI_Comm_check" returns (and possibly not even that). After that, it is > really the "group_of_processes_that_was_alive_at_some_point_in_the_past". > Since there are sometimes correlations in failures, this could happen even > if the initial failure is rare. An alternate form might be to have a > routine, collective over a communicator, that returns a new communicator > meeting some definition of "members were alive at some point during > construction". It wouldn't guarantee you could use it, but it would have > cleaner semantics.**** > > **** > > Bill**** > > **** > > On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote:**** > > ** ** > > I would like to argue for a simplified version of the proposal that covers > a large percentage of use-cases and resists adding new “features” for the > full-range of ABFT techniques. It is good if we have a more pragmatic view > and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps > just want to do something like this:**** > > **** > > for(… really long time …) {**** > > MPI_Comm_check(work_comm, &is_ok, &alive_group);**** > > if(!is_ok) {**** > > MPI_Comm_create_group(alive_group, …, &new_comm);**** > > // re-balance workload and use new_comm in rest of computation**** > > MPI_Comm_free(work_comm); // get rid of old comm**** > > work_comm = new_comm;**** > > } else {**** > > // continue computation using work_comm**** > > // if some proc failed in this iteration, roll back work done in this > iteration, go back to loop**** > > }**** > > }**** > > **** > > **** > > William Gropp**** > > Director, Parallel Computing Institute**** > > Deputy Director for Research**** > > Institute for Advanced Computing Applications and Technologies**** > > Paul and Cynthia Saylor Professor of Computer Science**** > > University of Illinois Urbana-Champaign**** > > **** > > **** > > **** > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft**** > > > > **** > > ** ** > > -- > Tony Skjellum, PhD > RunTime Computing Solutions, LLC > tony(a)runtimecomputing.com > direct: +1-205-314-3595 > cell: +1-205-807-4968**** > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968

1 0

Re: [Mpi3-ft] simplified FT proposal
by Sur, Sayantan 16 Jan '12

16 Jan '12

Hi Tony, In the example semantics you mentioned, are the "ops" required to return the same result on all processors? Although this doesn't change the API "op", but it does change completion semantics of almost all MPI ops. I hope I am correctly interpreting your message. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Anthony Skjellum Sent: Sunday, January 15, 2012 7:06 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal Everyone, I think we need to start from scratch. We should look for minimal fault-tolerant models that are achievable and useful. They may allow for a combination of faults (process and network), but in the end, as discussed in San Jose: FTBLOCK -------------- Start_Block(comm) op [normal MPI operation on communicator specified by Start_Block, or subset thereof] op op Test_Block(comm) Which either succeeds, or fails, on the whole list of operations, followed by ways to reconstruct communicators and add back processes (easily), provides for 3-level fault tolerant model a) Simply retry if the kind of error at the Test_Block is retryable b) Simply reconstruct the communicator, use algorithmic fault tolerance to get lost data, and retry the block c) drop back to 1 or more levels of checkpoint-restart. We can envision in this model an unrolling of work, in terms of a parameter N, if there is a lot of vector work, to allow granularity control as a function of fault environment. In some sense, a simpler model, that provides for efforts to detect a) by MPI b) allowing application monitor to assert failure asynchronously to this loop provides a more general ability to have coverage of faults, including but not limited to process faults and possible network faults. It changes the API very little. It motivates the use of buffers, not zero copy, to support the fact that you may to roll-back a series of operations, thereby revealing fault-free overhead directly. Start_Block and Test_Block are collective and synchronizing, such as barriers. Because we have uncertainty to within a message, multiple barriers (as mentioned by George Bosilca K to me in a sidebar at the meeting). We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we can. Once we have this stable intermediate form, we explore more options. I think it is important to recognize that the reconstruction step, including readding processes and making new communicators may mean smarter Join operations. It is clear we need to be able to treat failures during the recovery process, and use a second level loop, possibly bombing out to checkpoint, if we cannot make net progress on recovery because of unmodeled error issues. The testing part leverages all the learning so far, but needn't be restricted to modeled errors like process faults. There can be modeled and unmodeled faults. Based on what fault comes up, the user application then has to decide how hard a retry to do, whether just to add processes, whether just to retry the loop, whether to go to a checkpoint, whether to restart the APP. MPI could give advice, based on its understanding of the fault model, in terms of sufficient conditions for "working harder" vs "trying the easiest" for fault models it understands somewhat for a system. Now, the comments here are a synopsis of part of the side bars and open discussion we had in San Jose, distilled a bit. I want to know why we can't start with this, succeed with this, implement and test it, and having succeeded, do more in a future 3.y, y > x release, given user experience. I am not speaking to the choice of "killing all communicators" as with FT-MPI, or "just remaking those you need to remake." I think we need to resolve. Honestly, groups own the fault property, not communicators, and all groups held by communicators where the fault happened should be rebuilt, not all communicators... Let's argue on that. So, my suggestion is REBOOT the proposal with something along lines above, unless you all see this is no better. With kind regards, Tony On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan <sayantan.sur(a)intel.com<mailto:[email protected]>> wrote: Hi Bill, I am in agreement with your suggestion to have a collective over a communicator that returns a new communicator containing ranks "alive some point during construction". It provides cleaner semantics. The example was merely trying to utilize the new MPI_Comm_create_group API that the Forum is considering. MPI_Comm_check provides a method to form global consensus in that all ranks in comm did call it. It does not imply anything about current status of comm, or even the status "just before" the call returns. During the interval before the next call to MPI_Comm_check, it is possible that many ranks fail. However, the app/lib using MPI knows the point where everyone was alive. Thanks. === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of William Gropp Sent: Sunday, January 15, 2012 2:41 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] simplified FT proposal One concern that I have with fault tolerant proposals has to do with races in the specification. This is an area where users often "just want it to work" but getting it right is tricky. In the example here, the "alive_group" is really only that at some moment shortly before "MPI_Comm_check" returns (and possibly not even that). After that, it is really the "group_of_processes_that_was_alive_at_some_point_in_the_past". Since there are sometimes correlations in failures, this could happen even if the initial failure is rare. An alternate form might be to have a routine, collective over a communicator, that returns a new communicator meeting some definition of "members were alive at some point during construction". It wouldn't guarantee you could use it, but it would have cleaner semantics. Bill On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote: I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and resists adding new "features" for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this: for(... really long time ...) { MPI_Comm_check(work_comm, &is_ok, &alive_group); if(!is_ok) { MPI_Comm_create_group(alive_group, ..., &new_comm); // re-balance workload and use new_comm in rest of computation MPI_Comm_free(work_comm); // get rid of old comm work_comm = new_comm; } else { // continue computation using work_comm // if some proc failed in this iteration, roll back work done in this iteration, go back to loop } } William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com<mailto:[email protected]> direct: +1-205-314-3595 cell: +1-205-807-4968

1 0