May 2013 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL
by Wesley Bland 08 May '13

08 May '13

I plan to post the result of any discussions we have onto the ticket, but I know that most of this list isn't on the ticket so we'd miss out on most of the FT people. On May 8, 2013, at 10:23 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > It might be helpful to have this discussion on the ticket itself (or copy/paste it there), so that it will be saved for future reference. > > ~Jim. > > On 5/8/13 10:21 AM, Richard Graham wrote: >> I believe this received the "cold shoulder", and should be closed. >> >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aurï¿½lien Bouteiller >> Sent: Wednesday, May 08, 2013 11:10 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL >> >> This is a problem child. It changes the state of valid communicators by making operations operate on sparse communicators that contain holes (failed processes). This is difficult to implement and will have adverse performance hit, even outside failure cases. The functionality is duplicated with revoke/shrink, group_create, etc. I'm for killing this one, it has become irrelevant in the new context. >> >> Aurelien >> >> Le 7 mai 2013 ï¿½ 17:19, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : >> >>> author: jjhursey >>> >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/292 >>> >>> This proposal put forth a new function, MPI_COMM_KILL, that would exclude a remote rank from any further communication in all communicators in the MPI universe. >>> >>> One of the concerns of the forum is that it was starting to define semantics for failure scenarios other than fail-stop errors, specifically transient/intermittent failures. This was moved out of the previous RTS proposal because of this distinction. In the context of the current proposal, ULFM, this isn't really required anymore as the ULFM proposal states that once a process starts exhibiting failure, it is treated as fail-stop and should be excluded from further participation. >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> -- >> * Dr. Aurï¿½lien Bouteiller >> * Researcher at Innovative Computing Laboratory >> * University of Tennessee >> * 1122 Volunteer Boulevard, suite 309b >> * Knoxville, TN 37996 >> * 865 974 9375 >> >> >> >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #307: Behavior of MPI_INIT after failure to initialize
by Richard Graham 08 May '13

08 May '13

I do not thing there was any conclusion here - this is even true when one is not worrying about FT. What do you do when mpirun does not give you the number of process you asked for ? Before MPI_INIT we are not guaranteed to have more than a single "process". Rich -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Wesley Bland Sent: Wednesday, May 08, 2013 11:18 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Ticket #307: Behavior of MPI_INIT after failure to initialize Is there anything else from this ticket that we would find useful before killing it? For example, all of the text we have about MPI_INIT in ULFM is in an advice to implementors. Should we move anything to normative text? On May 8, 2013, at 10:06 AM, Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> wrote: > My memory is that this ticket has been combined, upgraded and included into ULFM. > > > Le 7 mai 2013 ï¿½ 17:40, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : > >> author: jjhursey >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/307 >> >> This ticket specifies the behavior of the MPI library after a during MPI_INIT. Basically, if there is a failure during INIT, the library should return an appropriate return code and calling any other MPI function (with a few exceptions) is erroneous and will trigger the error handler for MPI_COMM_WORLD, which by default is MPI_ERRORS_ARE_FATAL and cannot be changed in this instance. >> >> Again, this ticket has no discussion attached to it. I remember some discussions from the RTS proposal days, but I don't remember the results of any of those discussions. Can anyone else bring this back to light? >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL
by Jim Dinan 08 May '13

08 May '13

It might be helpful to have this discussion on the ticket itself (or copy/paste it there), so that it will be saved for future reference. ~Jim. On 5/8/13 10:21 AM, Richard Graham wrote: > I believe this received the "cold shoulder", and should be closed. > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aurï¿½lien Bouteiller > Sent: Wednesday, May 08, 2013 11:10 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL > > This is a problem child. It changes the state of valid communicators by making operations operate on sparse communicators that contain holes (failed processes). This is difficult to implement and will have adverse performance hit, even outside failure cases. The functionality is duplicated with revoke/shrink, group_create, etc. I'm for killing this one, it has become irrelevant in the new context. > > Aurelien > > Le 7 mai 2013 ï¿½ 17:19, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : > >> author: jjhursey >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/292 >> >> This proposal put forth a new function, MPI_COMM_KILL, that would exclude a remote rank from any further communication in all communicators in the MPI universe. >> >> One of the concerns of the forum is that it was starting to define semantics for failure scenarios other than fail-stop errors, specifically transient/intermittent failures. This was moved out of the previous RTS proposal because of this distinction. In the context of the current proposal, ULFM, this isn't really required anymore as the ULFM proposal states that once a process starts exhibiting failure, it is treated as fail-stop and should be excluded from further participation. >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL
by Richard Graham 08 May '13

08 May '13

I believe this received the "cold shoulder", and should be closed. -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aurï¿½lien Bouteiller Sent: Wednesday, May 08, 2013 11:10 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL This is a problem child. It changes the state of valid communicators by making operations operate on sparse communicators that contain holes (failed processes). This is difficult to implement and will have adverse performance hit, even outside failure cases. The functionality is duplicated with revoke/shrink, group_create, etc. I'm for killing this one, it has become irrelevant in the new context. Aurelien Le 7 mai 2013 ï¿½ 17:19, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : > author: jjhursey > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/292 > > This proposal put forth a new function, MPI_COMM_KILL, that would exclude a remote rank from any further communication in all communicators in the MPI universe. > > One of the concerns of the forum is that it was starting to define semantics for failure scenarios other than fail-stop errors, specifically transient/intermittent failures. This was moved out of the previous RTS proposal because of this distinction. In the context of the current proposal, ULFM, this isn't really required anymore as the ULFM proposal states that once a process starts exhibiting failure, it is treated as fail-stop and should be excluded from further participation. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375 _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #307: Behavior of MPI_INIT after failure to initialize
by Wesley Bland 08 May '13

08 May '13

Is there anything else from this ticket that we would find useful before killing it? For example, all of the text we have about MPI_INIT in ULFM is in an advice to implementors. Should we move anything to normative text? On May 8, 2013, at 10:06 AM, Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> wrote: > My memory is that this ticket has been combined, upgraded and included into ULFM. > > > Le 7 mai 2013 ï¿½ 17:40, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : > >> author: jjhursey >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/307 >> >> This ticket specifies the behavior of the MPI library after a during MPI_INIT. Basically, if there is a failure during INIT, the library should return an appropriate return code and calling any other MPI function (with a few exceptions) is erroneous and will trigger the error handler for MPI_COMM_WORLD, which by default is MPI_ERRORS_ARE_FATAL and cannot be changed in this instance. >> >> Again, this ticket has no discussion attached to it. I remember some discussions from the RTS proposal days, but I don't remember the results of any of those discussions. Can anyone else bring this back to light? >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL
by Wesley Bland 08 May '13

08 May '13

I would agree here. Especially as we now have REVOKE. Wesley On May 8, 2013, at 10:09 AM, Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> wrote: > This is a problem child. It changes the state of valid communicators by making operations operate on sparse communicators that contain holes (failed processes). This is difficult to implement and will have adverse performance hit, even outside failure cases. The functionality is duplicated with revoke/shrink, group_create, etc. I'm for killing this one, it has become irrelevant in the new context. > > Aurelien > > Le 7 mai 2013 ï¿½ 17:19, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : > >> author: jjhursey >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/292 >> >> This proposal put forth a new function, MPI_COMM_KILL, that would exclude a remote rank from any further communication in all communicators in the MPI universe. >> >> One of the concerns of the forum is that it was starting to define semantics for failure scenarios other than fail-stop errors, specifically transient/intermittent failures. This was moved out of the previous RTS proposal because of this distinction. In the context of the current proposal, ULFM, this isn't really required anymore as the ULFM proposal states that once a process starts exhibiting failure, it is treated as fail-stop and should be excluded from further participation. >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL
by Aurélien Bouteiller 08 May '13

08 May '13

This is a problem child. It changes the state of valid communicators by making operations operate on sparse communicators that contain holes (failed processes). This is difficult to implement and will have adverse performance hit, even outside failure cases. The functionality is duplicated with revoke/shrink, group_create, etc. I'm for killing this one, it has become irrelevant in the new context. Aurelien Le 7 mai 2013 ï¿½ 17:19, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : > author: jjhursey > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/292 > > This proposal put forth a new function, MPI_COMM_KILL, that would exclude a remote rank from any further communication in all communicators in the MPI universe. > > One of the concerns of the forum is that it was starting to define semantics for failure scenarios other than fail-stop errors, specifically transient/intermittent failures. This was moved out of the previous RTS proposal because of this distinction. In the context of the current proposal, ULFM, this isn't really required anymore as the ULFM proposal states that once a process starts exhibiting failure, it is treated as fail-stop and should be excluded from further participation. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [Mpi3-ft] Ticket #307: Behavior of MPI_INIT after failure to initialize
by Aurélien Bouteiller 08 May '13

08 May '13

My memory is that this ticket has been combined, upgraded and included into ULFM. Le 7 mai 2013 ï¿½ 17:40, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : > author: jjhursey > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/307 > > This ticket specifies the behavior of the MPI library after a during MPI_INIT. Basically, if there is a failure during INIT, the library should return an appropriate return code and calling any other MPI function (with a few exceptions) is erroneous and will trigger the error handler for MPI_COMM_WORLD, which by default is MPI_ERRORS_ARE_FATAL and cannot be changed in this instance. > > Again, this ticket has no discussion attached to it. I remember some discussions from the RTS proposal days, but I don't remember the results of any of those discussions. Can anyone else bring this back to light? > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

[Mpi3-ft] Ticket #336 Process Fault Tolerance Query Attribute
by Wesley Bland 07 May '13

07 May '13

author: jjhursey https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/336 This ticket provides a mechanism for an application to query the MPI implementation to determine if it provides the FT features in ULFM as some of the semantics can be provided only as non-functional stubs. I don't believe this was ever actually proposed at the forum because the ULFM ticket was postponed. This should probably remain in the pipeline and come back as a sub-ticket for ticket #323 later.

1 0

[Mpi3-ft] Ticket #324: Clarify MPI_ERRORS_ARE_FATAL scope of abort
by Wesley Bland 07 May '13

07 May '13

author: jjhursey https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 This ticket essentially links MPI_ERRORS_ARE_FATAL on a communicator to calling MPI_ABORT on the communicator, i.e. only the processes in that communicator are aborted, while other communicators could potentially remain functional. There was much discussion on the ticket about the scope of this change, and in the end the ticket has remained stagnant for about a year because of it, however I don't think that the changes here should be too controversial. According to the ticket, the main argument against it at the Japan meeting was that for some types of functions, there is not a request which can be used to provide error checking and therefore when an error occurs, the entire application would be forced to fall back to MPI_ERRORS_ARE_FATAL despite setting another error handler, therefore making FT difficult. Some alternate text was provided on the ticket.

1 0