January 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by Josh Hursey 24 Jan '12

24 Jan '12

It was also mentioned to me that the SHRINK mode also forces all messages to be dropped when the communicator is repaired. Shrinking will likely shift the ranks of processes in the communicator (to fill in the gaps) so matching will need to be reset, and messages would need to be dropped/drained. So it would no longer be possible for an application re-enable collectives and still have messages in-flight across the repair. At least not without further clarification on this point. -- Josh On Tue, Jan 24, 2012 at 2:41 PM, Graham, Richard L. <rlgraham(a)ornl.gov>wrote: > I will re-iterate what I said before. While this may one mode that apps > may want to use, it is not the only mode. In particular, this forces all > ranks in a communicator to know about the change, even if they have > implemented a "local" algorithm that does not need to know about all > failures. > > Rich > > On Jan 24, 2012, at 2:09 PM, Sur, Sayantan wrote: > > Hi Josh, > > Thanks for the crisp characterization of the proposal I was making. It is > correct. I was naturally thinking of the SHRINK mode, since it involves > least number of changes to MPI standard itself. Folks at the forum also had > similar thoughts (e.g. why does MPI_Comm_size() still return count > including failed procs). > > Cf. > http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT-MPI.pdf > > “4.2 FTMPI_COMM_MODE_SHRINK > > In this communicator mode, the ranks of MPI processes before and after > recovery might change, as well as the size of MPI COMM WORLD does change. > The appealing part of this communicator mode however is, that all functions > specified in MPI-1 and MPI-2 are still valid without any further > modification, since groups and communicators do not have wholes (sic) and > blank processes.” > > We can discuss further tomorrow as to whether we could go with SHRINK mode > (simplifying the proposal). From what I read in the paper, they report > being able to convert fault-tolerant master/worker to use both modes. > > Thanks! > > === > Sayantan Sur, Ph.D. > Intel Corp. > > From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org> [mailto: > mpi3-ft-bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey > Sent: Tuesday, January 24, 2012 10:39 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal > > First let me say that I greatly appreciate the effort of Sayantan and > others to push us towards considering alternative techniques, and > stimulating discussion about design decisions. This is exactly the type of > discussion that needs to occur, and the working group is the most > appropriate place to have it. > > > One of the core suggestions of Sayantan's proposal is the switch from > (using FT-MPI's language) a model like BLANK to a model like SHRINK. I > think many of the other semantics are derived from this core shift, so we > should probably focus the discussion on this point earlier in our > conversation. > > > The current RTS proposal allows for a communicator to contain failed > processes and continue to be used for all operations, including > collectives, after acknowledging them. This matches closely to FT-MPI's > BLANK mode. The user can use MPI_Comm_split() to get the equivalent of > SHRINK if they need it. > > The suggested modification allows for only/primarily a SHRINK-like mode in > order to have full functionality in the communicator. As discussed on the > previous call, one can get the BLANK mode by adding a library on top of MPI > that virtualizes the communicators to create shadow communicators. The > argument for the SHRINK mode is that it is -easier- to pass/explain. > > The reason we chose BLANK was derived from the literature reviewed, code > examples available, and feedback from application groups. >From which there > seemed to be a strong demand for the BLANK mode. In fact, I had a difficult > time finding good use cases for the SHIRNK mode (I'm still looking though). > Additionally, a BLANK mode seems also to make it easier to reason about > process recovery. To reason about process recovery (something like FT-MPI's > REBUILD mode) one needs to be able to reason about the missing processes > without changing the identities of the existing processes, which can be > difficult in a SHRINK mode. So from this review it seemed that there was an > application demand for a BLANK-like mode for the RTS proposal. > > In light of this background, it is concerning to me to advise these > application users that MPI will not provide the functionality they require, > but they have to depend upon a non-standard, third-party library because we > shied away from doing the right thing by them. This background is advised > from my review of the state of the art, but others may have alternative > evidence/commentary to present as well that could sway the discussion. It > just seems like a weak argument that we should do the easy thing at the > expense of doing the right thing by the application community. > > > I certainly meant this email to stimulate conversation for the > teleconference tomorrow. In particular, I would like those on the list with > experience building ABFT/Natural FT applications/libraries (UTK?) to > express their perspective on this topic. Hopefully they can help guide us > towards the right solution, which might just be a SHRINK-like mode. > > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by Graham, Richard L. 24 Jan '12

24 Jan '12

I will re-iterate what I said before. While this may one mode that apps may want to use, it is not the only mode. In particular, this forces all ranks in a communicator to know about the change, even if they have implemented a "local" algorithm that does not need to know about all failures. Rich On Jan 24, 2012, at 2:09 PM, Sur, Sayantan wrote: Hi Josh, Thanks for the crisp characterization of the proposal I was making. It is correct. I was naturally thinking of the SHRINK mode, since it involves least number of changes to MPI standard itself. Folks at the forum also had similar thoughts (e.g. why does MPI_Comm_size() still return count including failed procs). Cf. http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT-MPI.pdf ï¿½4.2 FTMPI_COMM_MODE_SHRINK In this communicator mode, the ranks of MPI processes before and after recovery might change, as well as the size of MPI COMM WORLD does change. The appealing part of this communicator mode however is, that all functions specified in MPI-1 and MPI-2 are still valid without any further modification, since groups and communicators do not have wholes (sic) and blank processes.ï¿½ We can discuss further tomorrow as to whether we could go with SHRINK mode (simplifying the proposal). From what I read in the paper, they report being able to convert fault-tolerant master/worker to use both modes. Thanks! === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Tuesday, January 24, 2012 10:39 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal First let me say that I greatly appreciate the effort of Sayantan and others to push us towards considering alternative techniques, and stimulating discussion about design decisions. This is exactly the type of discussion that needs to occur, and the working group is the most appropriate place to have it. One of the core suggestions of Sayantan's proposal is the switch from (using FT-MPI's language) a model like BLANK to a model like SHRINK. I think many of the other semantics are derived from this core shift, so we should probably focus the discussion on this point earlier in our conversation. The current RTS proposal allows for a communicator to contain failed processes and continue to be used for all operations, including collectives, after acknowledging them. This matches closely to FT-MPI's BLANK mode. The user can use MPI_Comm_split() to get the equivalent of SHRINK if they need it. The suggested modification allows for only/primarily a SHRINK-like mode in order to have full functionality in the communicator. As discussed on the previous call, one can get the BLANK mode by adding a library on top of MPI that virtualizes the communicators to create shadow communicators. The argument for the SHRINK mode is that it is -easier- to pass/explain. The reason we chose BLANK was derived from the literature reviewed, code examples available, and feedback from application groups. >From which there seemed to be a strong demand for the BLANK mode. In fact, I had a difficult time finding good use cases for the SHIRNK mode (I'm still looking though). Additionally, a BLANK mode seems also to make it easier to reason about process recovery. To reason about process recovery (something like FT-MPI's REBUILD mode) one needs to be able to reason about the missing processes without changing the identities of the existing processes, which can be difficult in a SHRINK mode. So from this review it seemed that there was an application demand for a BLANK-like mode for the RTS proposal. In light of this background, it is concerning to me to advise these application users that MPI will not provide the functionality they require, but they have to depend upon a non-standard, third-party library because we shied away from doing the right thing by them. This background is advised from my review of the state of the art, but others may have alternative evidence/commentary to present as well that could sway the discussion. It just seems like a weak argument that we should do the easy thing at the expense of doing the right thing by the application community. I certainly meant this email to stimulate conversation for the teleconference tomorrow. In particular, I would like those on the list with experience building ABFT/Natural FT applications/libraries (UTK?) to express their perspective on this topic. Hopefully they can help guide us towards the right solution, which might just be a SHRINK-like mode. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by Sur, Sayantan 24 Jan '12

24 Jan '12

Hi Josh, Thanks for the crisp characterization of the proposal I was making. It is correct. I was naturally thinking of the SHRINK mode, since it involves least number of changes to MPI standard itself. Folks at the forum also had similar thoughts (e.g. why does MPI_Comm_size() still return count including failed procs). Cf. http://www.netlib.org/utk/people/JackDongarra/PAPERS/isc2004-FT-MPI.pdf "4.2 FTMPI_COMM_MODE_SHRINK In this communicator mode, the ranks of MPI processes before and after recovery might change, as well as the size of MPI COMM WORLD does change. The appealing part of this communicator mode however is, that all functions specified in MPI-1 and MPI-2 are still valid without any further modification, since groups and communicators do not have wholes (sic) and blank processes." We can discuss further tomorrow as to whether we could go with SHRINK mode (simplifying the proposal). From what I read in the paper, they report being able to convert fault-tolerant master/worker to use both modes. Thanks! === Sayantan Sur, Ph.D. Intel Corp. From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Josh Hursey Sent: Tuesday, January 24, 2012 10:39 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal First let me say that I greatly appreciate the effort of Sayantan and others to push us towards considering alternative techniques, and stimulating discussion about design decisions. This is exactly the type of discussion that needs to occur, and the working group is the most appropriate place to have it. One of the core suggestions of Sayantan's proposal is the switch from (using FT-MPI's language) a model like BLANK to a model like SHRINK. I think many of the other semantics are derived from this core shift, so we should probably focus the discussion on this point earlier in our conversation. The current RTS proposal allows for a communicator to contain failed processes and continue to be used for all operations, including collectives, after acknowledging them. This matches closely to FT-MPI's BLANK mode. The user can use MPI_Comm_split() to get the equivalent of SHRINK if they need it. The suggested modification allows for only/primarily a SHRINK-like mode in order to have full functionality in the communicator. As discussed on the previous call, one can get the BLANK mode by adding a library on top of MPI that virtualizes the communicators to create shadow communicators. The argument for the SHRINK mode is that it is -easier- to pass/explain. The reason we chose BLANK was derived from the literature reviewed, code examples available, and feedback from application groups. >From which there seemed to be a strong demand for the BLANK mode. In fact, I had a difficult time finding good use cases for the SHIRNK mode (I'm still looking though). Additionally, a BLANK mode seems also to make it easier to reason about process recovery. To reason about process recovery (something like FT-MPI's REBUILD mode) one needs to be able to reason about the missing processes without changing the identities of the existing processes, which can be difficult in a SHRINK mode. So from this review it seemed that there was an application demand for a BLANK-like mode for the RTS proposal. In light of this background, it is concerning to me to advise these application users that MPI will not provide the functionality they require, but they have to depend upon a non-standard, third-party library because we shied away from doing the right thing by them. This background is advised from my review of the state of the art, but others may have alternative evidence/commentary to present as well that could sway the discussion. It just seems like a weak argument that we should do the easy thing at the expense of doing the right thing by the application community. I certainly meant this email to stimulate conversation for the teleconference tomorrow. In particular, I would like those on the list with experience building ABFT/Natural FT applications/libraries (UTK?) to express their perspective on this topic. Hopefully they can help guide us towards the right solution, which might just be a SHRINK-like mode. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Choosing a BLANK or SHRINK model for the RTS proposal
by Josh Hursey 24 Jan '12

24 Jan '12

First let me say that I greatly appreciate the effort of Sayantan and others to push us towards considering alternative techniques, and stimulating discussion about design decisions. This is exactly the type of discussion that needs to occur, and the working group is the most appropriate place to have it. One of the core suggestions of Sayantan's proposal is the switch from (using FT-MPI's language) a model like BLANK to a model like SHRINK. I think many of the other semantics are derived from this core shift, so we should probably focus the discussion on this point earlier in our conversation. The current RTS proposal allows for a communicator to contain failed processes and continue to be used for all operations, including collectives, after acknowledging them. This matches closely to FT-MPI's BLANK mode. The user can use MPI_Comm_split() to get the equivalent of SHRINK if they need it. The suggested modification allows for only/primarily a SHRINK-like mode in order to have full functionality in the communicator. As discussed on the previous call, one can get the BLANK mode by adding a library on top of MPI that virtualizes the communicators to create shadow communicators. The argument for the SHRINK mode is that it is -easier- to pass/explain. The reason we chose BLANK was derived from the literature reviewed, code examples available, and feedback from application groups. From which there seemed to be a strong demand for the BLANK mode. In fact, I had a difficult time finding good use cases for the SHIRNK mode (I'm still looking though). Additionally, a BLANK mode seems also to make it easier to reason about process recovery. To reason about process recovery (something like FT-MPI's REBUILD mode) one needs to be able to reason about the missing processes without changing the identities of the existing processes, which can be difficult in a SHRINK mode. So from this review it seemed that there was an application demand for a BLANK-like mode for the RTS proposal. In light of this background, it is concerning to me to advise these application users that MPI will not provide the functionality they require, but they have to depend upon a non-standard, third-party library because we shied away from doing the right thing by them. This background is advised from my review of the state of the art, but others may have alternative evidence/commentary to present as well that could sway the discussion. It just seems like a weak argument that we should do the easy thing at the expense of doing the right thing by the application community. I certainly meant this email to stimulate conversation for the teleconference tomorrow. In particular, I would like those on the list with experience building ABFT/Natural FT applications/libraries (UTK?) to express their perspective on this topic. Hopefully they can help guide us towards the right solution, which might just be a SHRINK-like mode. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ Noon EST
by Josh Hursey 24 Jan '12

24 Jan '12

Reminder that we have a teleconf tomorrow, Wednesday, at Noon EST. Date: January 25, 2012 Time: Noon EST/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Suggested modifications to RTS proposal (Sayantan) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/alt_simplified_rts_pro… http://lists.mpi-forum.org/mpi3-ft/2012/01/0959.php - Tony's alternative model (Tony) - Do you still need this time? - Other items? Additional Agenda Items [as time permits]: - Run-Through Stabilization Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - Piggybacking (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - MPI Forum notes discussion (?) http://lists.mpi-forum.org/mpi3-ft/2011/07/0805.php - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php - FT Query Interface http://lists.mpi-forum.org/mpi3-ft/2011/06/0737.php Next Meetings: * Jan. 25, 2012 (Teleconf) * Feb. 1, 2012 (Teleconf) * Feb. 8, 2012 (Teleconf - As needed) * Feb. 15, 2012 (Teleconf) * Feb. 22, 2012 (Teleconf - As needed) * Feb. 29, 2012 (Teleconf) * March 5, 2012 (MPI Forum Meeting) * March 14, 2012 (Teleconf) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] UAB Model Discussion
by Josh Hursey 23 Jan '12

23 Jan '12

The list of upcoming teleconfs can be found on the following website: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/FaultToleranceWikiPage Feb 1 will be a teleconf. Feb 8 is currently 'as needed' and will probably be decided the week before (during the Feb 1 meeting). -- Josh On Mon, Jan 23, 2012 at 4:42 PM, Anthony Skjellum <tony(a)cis.uab.edu> wrote: > Josh, on a related note, is there a con call on Feb 1 or Feb 8? by Feb 8, > we will have our write-up > on the proposed alternative model done... > > Regards, > Tony > > > -- > Anthony Skjellum, PhD > Professor and Chair > Dept. of Computer and Information Sciences > Director, UAB Center for Information Assurance and Joint Forensics > Research ("The Center") > University of Alabama at Birmingham > +1-(205)934-8657; FAX: +1- (205)934-5473 > > ___________________________________________ > CONFIDENTIALITY: This e-mail and any attachments are confidential and > may be privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person, > use it for any purpose or store or copy the information in any medium. > > Please consider the environment before printing this e-mail > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] UAB Model Discussion
by Anthony Skjellum 23 Jan '12

23 Jan '12

Josh, on a related note, is there a con call on Feb 1 or Feb 8? by Feb 8, we will have our write-up on the proposed alternative model done... Regards, Tony ----- Original Message ----- From: "Josh Hursey" <jjhursey(a)open-mpi.org> To: mpi3-rma(a)lists.mpi-forum.org, "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Sent: Monday, January 23, 2012 3:33:25 PM Subject: [Mpi3-ft] Fault Tolerance & RMA Discussion (Cross posted to both the RMA and FT MPI-3 listservs) During the FT plenary session at the Jan. MPI Forum meeting it was recommended that some of the members of the FT group and the RMA group have a meeting to hash out the precise details of the FT semantics for the RMA chapter. So I would like to facilitate such a discussion, preferability in the next week (so we have time to fine tune things before the next forum meeting). In general, we are trying to answer the question "How should RMA operations behave when a process failure occurs?" The feeling seemed to be that the current approach is ok (invalidating the window, forcing recreation/validation), but the statement that the memory exposed in the window is 'undefined' seemed excessive. The suggestion was to change the wording to something like "Only the memory associated with a window that was targeted by an operation that modified it is undefined after process failure in the group associated with the window." This lead to a considerable amount of debate in the meeting, so it was suggested that we take the discussion offline. Below is a link to a doodle poll to find a good time for a teleconf. If you are interested in participating in this discussion, please fill this poll out by 2 PM Eastern on Wed. Jan 25 so we can set the date/time. http://www.doodle.com/vd33va5h8iankega Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

[Mpi3-ft] Fault Tolerance & RMA Discussion
by Josh Hursey 23 Jan '12

23 Jan '12

(Cross posted to both the RMA and FT MPI-3 listservs) During the FT plenary session at the Jan. MPI Forum meeting it was recommended that some of the members of the FT group and the RMA group have a meeting to hash out the precise details of the FT semantics for the RMA chapter. So I would like to facilitate such a discussion, preferability in the next week (so we have time to fine tune things before the next forum meeting). In general, we are trying to answer the question "How should RMA operations behave when a process failure occurs?" The feeling seemed to be that the current approach is ok (invalidating the window, forcing recreation/validation), but the statement that the memory exposed in the window is 'undefined' seemed excessive. The suggestion was to change the wording to something like "Only the memory associated with a window that was targeted by an operation that modified it is undefined after process failure in the group associated with the window." This lead to a considerable amount of debate in the meeting, so it was suggested that we take the discussion offline. Below is a link to a doodle poll to find a good time for a teleconf. If you are interested in participating in this discussion, please fill this poll out by 2 PM Eastern on Wed. Jan 25 so we can set the date/time. http://www.doodle.com/vd33va5h8iankega Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization
by Rolf Rabenseifner 23 Jan '12

23 Jan '12

George and all, yes, to use collective routines instead of the currently non-collective definition of MPI_Failhandler_set_mode makes more sense than doing something at MPI_Init time. Therefore, I withdraw my idea of extending MPI_INIT_THREAD. Best regards Rolf ----- Original Message ----- > From: "George Bosilca" <bosilca(a)eecs.utk.edu> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Terry D. Dontje" <terry.dontje(a)oracle.com>, "Josh Hursey" <jjhursey(a)open-mpi.org>, "Bronis R. de Supinski" > <bronis(a)llnl.gov>, "Pavan Balaji" <balaji(a)mcs.anl.gov>, "MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group" <mpi3-ft(a)lists.mpi-forum.org> > Sent: Wednesday, January 18, 2012 8:40:50 PM > Subject: Re: MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at MPI initialization > On Jan 18, 2012, at 14:06 , Rolf Rabenseifner wrote: > > > To make MPI_Failhandler_set_mode collective is another choice. > > Collective over what? All connected processes? > > As you pointed out, collective over what? Per communicator? Per > MPI_COMM_WORLD? We should forget about the all connected processes, as > the current MPI standard have a very careless definition of it. > > > What is with processes that are spawned after > > MPI_Failhandler_set_mode? > > I don't think this function is needed at all. The setting of the > failure handler (MPI_Comm_set_failhandler) should be collective over > the communicator provided as argument. In the case of spawned or > connected processes, it is the upper layer responsibility to set the > failure handler to what make sense for the application stack (either > by creating an intra-comm and setting the failure handler there or we > might want to extend the logic of the MPI_Comm_set_failhandler to > cover inter-comms). > > > The other idea is, to allow setting special options before > > MPI_Init/MPI_Init_thread. > > I think it makes more sense if these options are somehow global … at > least to the MPI_COMM_WORLD. Maybe via the command line? > > > The only requirement is, that the FT proposal is consistent > > with the rest of MPI. > > We already have some inconsistencies in the current standard, it will > be abusing to require from the FT proposal what the remaining of the > MPI standard fails to deliver. Don't get me wrong here, I'm not saying > we should let it loose either. Some level of decency should be > enforced. > > george. > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "George Bosilca" <bosilca(a)eecs.utk.edu> > >> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working > >> Group" <mpi3-ft(a)lists.mpi-forum.org> > >> Cc: "Terry D. Dontje" <terry.dontje(a)oracle.com>, "Josh Hursey" > >> <jjhursey(a)open-mpi.org>, "Rolf Rabenseifner" > >> <rabenseifner(a)hlrs.de>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, > >> "Pavan Balaji" <balaji(a)mcs.anl.gov> > >> Sent: Wednesday, January 18, 2012 4:26:43 PM > >> Subject: Re: MPI_INIT_THREAD and MPI_Failhandler_set/get_mode at > >> MPI initialization > >> I concur with the previous statements. As Rolf highlighted it in > >> his > >> email, one of the reasons of this new proposal is to fix the > >> "unclear" > >> collective behavior of MPI_Failhandler_set_mode. I don't see the > >> unclearness, and here are two of my reasons. > >> > >> > >> 1. There is no reason to have such a function > >> (MPI_Failhandler_set_mode), setting of the fail handler should > >> ALWAYS > >> be collective, otherwise the entire purpose of the fail handler is > >> annihilated. > >> > >> > >> 2. If no collective behavior is required (meaning the software > >> stack > >> doesn't have to be rebuild in a collective way), then the fail > >> handler > >> is a clear overkill. A saner and more clear behavior can be > >> obtained > >> by using local Error handler with carefully crafted requests (as an > >> example a non-blocking, never to be matched, request on a duplicate > >> of > >> MPI_COMM_WORLD can do the trick). > >> > >> > >> george. > >> > >> > >> > >> On Jan 18, 2012, at 09:56 , Josh Hursey wrote: > >> > >> > >> I like the motivation of the proposal, but I think Terry has a good > >> point. It seems a bit like a hack to repurpose the > >> required/provided > >> arguments to achieve semantic assertions. I would almost prefer > >> some > >> other functionality that must be called before MPI_Init{_thread} > >> that > >> would explicitly set these options. That starts to sound like the > >> assertion ticket that Terry mentioned. So maybe they can be merged > >> or > >> revised. > >> > >> > >> I am also a bit concerned about having conditional semantics in the > >> MPI standard. Though the FT proposal is founded in the condition > >> that > >> the semantics are only meaningful when the error handler is not > >> ARE_FATAL, which is conditional. So I am a bit torn on this point. > >> > >> > >> One thing that your proposal should clearly specify is whether the > >> specified bits must be set to the same value at all > >> processes/threads. > >> Additionally, what if two MPI_COMM_WORLDs connect/accept but have > >> different bits set? Does that restrict how these two world can > >> interact? This was one of the problems posed for the > >> Failhandler_set_mode() semantics that still needs to be addressed > >> here, but in a more general sense. > >> > >> > >> So I think it is an interesting proposal worth considering further. > >> Setting options like 'enable FT' at initialization time (or just > >> before initialization time) might allow the MPI implementation to > >> optimize the library appropriately during setup (choosing different > >> components or algorithms). It might be worth looking at the > >> assertions > >> proposal to see if there is a viable alternative solution there > >> that > >> would achieve the same goals as this proposal without repurposing > >> the > >> required/provided arguments of MPI_Init_thread. > >> > >> > >> -- Josh > >> > >> > >> > >> On Wed, Jan 18, 2012 at 9:32 AM, TERRY DONTJE < > >> terry.dontje(a)oracle.com > wrote: > >> > >> > >> > >> I think the idea is worthwhile but it really smells similar to the > >> defunct assertion ticket. I really find piggy-backing the ft, > >> cancel > >> and any_source modes onto the required/provided bits a little > >> unpleasing to my senses. The reason I am displeased with the > >> proposal > >> is it seems to slightly open a door to give an application the > >> ability > >> to give hints and if we are going to do that we might as well open > >> the > >> door fully and allow vendor specific hints. Doing the latter will > >> require more than the require/provided bits. > >> > >> The above aside, if the proposal is passed I guess my only other > >> comment is the of moving the MPI_INIT_THREAD & MPI_QUERY_THREAD to > >> 8.7 > >> (startup) seems odd to me. I guess I can see the reasoning of > >> moving > >> the interface to the Startup section but then the threadsafety > >> portion > >> of 12.4.3 section seems to stick out strangely IMO. > >> > >> --td > >> > >> > >> > >> On 1/17/2012 5:31 AM, Rolf Rabenseifner wrote: > >> > >> Dear committees of - FT, - MPI_Init --> 8. Environmental > >> Management, - > >> MPI_Init_thread --> 12. External Interfaces. Before discussing > >> details, I would like to get a clear answer whether you believe > >> that > >> the proposal below is a good or bad idea. As already mentioned at > >> the > >> Jan. 2012 meeting, I would like to propose that the FT group may > >> substitute the unclear collective behavior of > >> MPI_Failhandler_set_mode > >> by adding the mode to the MPI initialization. For this, I added a > >> proposal to slide 4 in > >> MPI_Forum_Overview_MPI-3.0_Jan2012_action-items.ppt (see my > >> previous > >> mail to the MPI-Forum list) - If appropriate, a new ticket that > >> enhances MPI_INIT_THREAD -- Required and provided as “bit vector” > >> of > >> "bit-wise OR" of required_/provided_threadsafety | > >> required_/provided_ft_mode | required_/provided_cancel_mode | > >> required_/provided_any_source_mode -- New mask-constants > >> MPI_THREAD_MASK, MPI_FT_MASK, MPI_CANCEL_MASK, MPI_ANY_SOURCE_MASK > >> -- > >> With existing values for required_/provided_threadsafety > >> MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, MPI_THREAD_SERIALIZED, and > >> MPI_THREAD_MULTIPLE. -- With new values for - > >> required_/provided_ft_mode = MPI_FT_NONE=0, or > >> MPI_FT_FAILHANDLER_MODE_ALL≠0, or MPI_FT_FAILHANDLER_MODE_SUBSET≠0, > >> or > >> - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED=0, or > >> MPI_NO_CANCEL≠0 - required_/provided_any_source_mode = > >> MPI_ANY_SOURCE_ALLOWED=0, or MPI_NO_ANY_SOURCE≠0 -- Values must be > >> set > >> identical for all processes in an MPI_COMM_WORLD - It is easier to > >> relax about this in further versions of MPI than to relax already > >> now > >> and to restrict later as now done in ticket #222 for > >> required_/provided_threadsafety - For each of the for "variables", > >> a > >> different decision can be done. -- At least for > >> required_/provided_cancel_mode and ...any_source_mode, I would > >> require > >> that the provided value must be identical to the required value. > >> Reason: Internally, the value can be ignored. -- For > >> required_/provided_ft_mode, I would recommend to allow that > >> provided_ft_mode must be - identical to required_ft_mode or > >> MPI_FT_NONE - and the same in all processes. -- MPI_INIT_THREAD & > >> MPI_QUERY_THREAD moves - from 12.4.3 (External Interfaces) - to 8.7 > >> (Startup) - but explanations to ...threadsafety | ...ft_mode | > >> ...cancel_mode | ...any_source_mode are kept or written in the > >> appropriate sections 12.4.3, new 17.5 (FT Environm.), 3.8.4 > >> (Cancel), > >> 3.2.4 (Blocking Receive) -- A call to MPI_INIT is identical to > >> MPI_INIT_THREAD with - the rules in 12.4.3 about > >> required_/provided_threadsafety - required_/provided_ft_mode = > >> MPI_FT_NONE, - required_/provided_cancel_mode = MPI_CANCEL_ALLOWED, > >> - > >> required_/provided_any_source_mode = MPI_ANY_SOURCE_ALLOWED -- This > >> ticket would have the following properties: - It is clearly > >> source-code and ABI backward compatible, because - the values of > >> MPI_THREAD_SINGLE, _FUNNELED, ... need not to be changed and > >> MPI_THREAD_SINGLE need not to be zero; - the values representing > >> the > >> current MPI-2.2 quality are set to zero: MPI_FT_NONE=0, > >> MPI_CANCEL_ALLOWED=0, and MPI_ANY_SOURCE_ALLOWED=0 - It is very > >> unlikely that an implementation has used more than 24 different > >> bits > >> in these 4 integer constants MPI_THREAD_SINGLE, ... _MULTIPLE. > >> Therefore MPI_THREAD_MASK would have a maximum of 24 bits. Enough > >> room > >> for the 4 bits needed together for the other three ..._MASKs. - FT > >> can > >> be switched on or off at the MPI initialization and is switched off > >> in > >> unchanged applications. Therefore no backward-compatibility-problem > >> with a modified behavior of the default error handlers when FT is > >> switched on. - Normally there should be enough bit-space for > >> further > >> decisions at MPI initialization. - The decisions about the cancel > >> and > >> any_source values would be done in different tickets - FT quality > >> is > >> optional if we add the rule that provided_ft_mode may be identical > >> to > >> required_ft_mode ***or*** MPI_FT_NONE - This rule can be changed in > >> a > >> further version of MPI without backward-compatibility-problems. I > >> would like to get a reply from - the FT group - the chapter > >> committee > >> of MPI_Init --> 8. Environmental Manag. George Bosilca(c), Josh > >> Hursey, Terry Dontje - the chapter committee of MPI_Init_thread --> > >> 12. External Interf. Bronis R. de Supinski(c), Pavan Balaji Before > >> discussing details, I would like to get a clear answer whether you > >> believe that this is a good or bad idea. Best regards Rolf > >> > >> -- > >> > >> <Mail Attachment.gif> > >> > >> > >> > >> > >> Terry D. Dontje | Principal Software Engineer > >> > >> Developer Tools Engineering | +1.781.442.2631 > >> Oracle - Performance Technologies > >> 95 Network Drive, Burlington, MA 01803 > >> Email terry.dontje(a)oracle.com > >> > >> > >> > >> > >> > >> > >> > >> -- > >> Joshua Hursey > >> Postdoctoral Research Associate > >> Oak Ridge National Laboratory > >> http://users.nccs.gov/~jjhursey > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

[Mpi3-ft] MPI Forum Plenary Notes - Jan 2012
by Josh Hursey 19 Jan '12

19 Jan '12

Attached are the notes that I was able to accumulate from the FT working group plenary session during the Jan. 2012 MPI Forum meeting. Thanks to everyone that sent me notes to contribute to this effort. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0