May 2013 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Q: Revisit meeting time?
by Thomas Naughton 16 May '13

16 May '13

Hello, Several of us at ORNL were curious if it would be possible to revise the meeting time for the bi-weekly telecons. We have a recurring noon/lunch event that conflicts with the MPI-FT Working Group call. We would like to attend on a more regular basis but this conflict limits regular attendance. I know schedules are difficult to change but we wanted to ask and see. Respectfully, --tjn _________________________________________________________________________ Thomas Naughton naughtont(a)ornl.gov Research Associate (865) 576-4184

1 0

[Mpi3-ft] Conference Call May 15, noon EDT
by Aurélien Bouteiller 14 May '13

14 May '13

Dear WG members, This is a reminder that according to our planning, we are having our regular phone meeting tomorrow at noon EDT. Date: May 15, Time: Noon EDT/New York Dial-in information: 218-339-4600 Code: 623998# Next Meetings: * TBD??? May 29, 2013 * June 26, 2013 Items: * Next meeting schedule and preparation * Fn classification efforts * Ticket cleanup/revisit As a reminder, I attach highlights from notes from last meeting minutes Directions * Doing the interface coverage of ULFM (former object classification, now fn classification), we found it is difficult to follow if a particular function is local, collective, collective on a group, or just has some remote action (or needs a matching call). In some functions, the description is explicit, but buried deep in the text and difficult to find. In some other, it is never explicit, and for some it is hard to infer from context. We propose to spin out a separate, non FT related ticket, on adding a clarification in a systematic way to all functions, wether they are local, need a matching call, are collective, and if so on what realm they are collective. This is not specifically FT related, and is supposed to live its own life. We would like time for this plenary in the next forum. * I/O section has not been considered in a long time. It seems fine and we don't see a need to change it yet, but we may want to recheck with Quincy and Mohamed that we are still on the same page. * Argonne plans on mpich prototype implementation * We have a number of old tickets, from MPI-3 efforts. Many are still open. Most probably they need to be closed as they have been voted down, or have become irrelevant. Some may be of some interest to continue pursuing. Wesley is to revisit them and make a short report before next FTWG confcall. * We still need to rebase from the MPI-3 document (we are still forked from MPI-2 repo). Time needed during the next forum meeting: * Some plenary time for fn type clarification * Some RMA+FT WG time -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL
by Wesley Bland 14 May '13

14 May '13

You're right that it's not exactly duplicated functionality, but the scope of the ticket is outside of the supported set of errors. We only deal with fail-stop and this implies something more nefarious or at least less predictable. Wesley On May 13, 2013, at 10:09 PM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > While I'm not supporting the functionality proposed on this ticket, I don't see how this ticket is a duplicate of MPI_COMM_REVOKE. This function allows any process to propagate the exclusion of any other process from the known universe, based on local information. Such behavior is difficult to implement in ULFM as it will require an agreement, thus a synchronizing operation. The only similarity I see with MPI_COMM_REVOKE is the way the kill message is propagated (operation with a global scope but without global call). > > George. > > On May 8, 2013, at 11:27 , Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> I plan to post the result of any discussions we have onto the ticket, but I know that most of this list isn't on the ticket so we'd miss out on most of the FT people. >> >> On May 8, 2013, at 10:23 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >> >>> It might be helpful to have this discussion on the ticket itself (or copy/paste it there), so that it will be saved for future reference. >>> >>> ~Jim. >>> >>> On 5/8/13 10:21 AM, Richard Graham wrote: >>>> I believe this received the "cold shoulder", and should be closed. >>>> >>>> -----Original Message----- >>>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aurï¿½lien Bouteiller >>>> Sent: Wednesday, May 08, 2013 11:10 AM >>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>>> Subject: Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL >>>> >>>> This is a problem child. It changes the state of valid communicators by making operations operate on sparse communicators that contain holes (failed processes). This is difficult to implement and will have adverse performance hit, even outside failure cases. The functionality is duplicated with revoke/shrink, group_create, etc. I'm for killing this one, it has become irrelevant in the new context. >>>> >>>> Aurelien >>>> >>>> Le 7 mai 2013 ï¿½ 17:19, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : >>>> >>>>> author: jjhursey >>>>> >>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/292 >>>>> >>>>> This proposal put forth a new function, MPI_COMM_KILL, that would exclude a remote rank from any further communication in all communicators in the MPI universe. >>>>> >>>>> One of the concerns of the forum is that it was starting to define semantics for failure scenarios other than fail-stop errors, specifically transient/intermittent failures. This was moved out of the previous RTS proposal because of this distinction. In the context of the current proposal, ULFM, this isn't really required anymore as the ULFM proposal states that once a process starts exhibiting failure, it is treated as fail-stop and should be excluded from further participation. >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> -- >>>> * Dr. Aurï¿½lien Bouteiller >>>> * Researcher at Innovative Computing Laboratory >>>> * University of Tennessee >>>> * 1122 Volunteer Boulevard, suite 309b >>>> * Knoxville, TN 37996 >>>> * 865 974 9375 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #324: Clarify MPI_ERRORS_ARE_FATAL scope of abort
by Aurélien Bouteiller 13 May '13

13 May '13

They are, but not in the scope of the ticket, which is only to define the propagation range of abort (restricting it to the comm on which it is invoked, rather than all processes). The items discussed in the ticket comments are important, but unrelated to the ticket itself. The simple answer to the problem of "my free request bring the entire system down" is "do not free requests that are not completed, it will trigger the wrong error handler". If really this is an important functionality that needs to be supported, I advocate dealing with it in a separate ticket. The ticket at hand clarifies -only- the scope of abort propagation (on the comm), this distinction would clarify what communicator abort should be called on for orphaned requests; two different issues. For buffered sends, no error should be triggered. The buffered send has succeeded (in the sense that the expected specification on the send buffer has been met). It is a case of lazy error detection that is perfectly valid in the context of ULFM (just like it may appear that some process has been reached by a bcast, from the perspective of the root, but was in fact dead). An error will be returned in a subsequent operation. Remember that no correctness is lost here, the scenario where the message is fully received -just before- the receiver dies is isomorphic in term of "returned errors", so the application must already deal with such a case. Aurelien. Le 13 mai 2013 ï¿½ 23:01, George Bosilca <bosilca(a)icl.utk.edu> a ï¿½crit : > Dave points are entirely valid and represent a subtle [corner-]case in the standard. Orphaned non-completed requests, in the sense that the request was freed (MPI_Request_free) and the associated communicator was freed as well (MPI_Comm_free) are defined as raising errors on MPI_COMM_WORLD. Thus, the scope of the request become global, and a fault on such particular requests will bring down the entire MPI_COMM_WORLD (which is against the original scope of the ticket, at least the first part). > > George. > > > On May 13, 2013, at 12:05 , Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> After looking at this ticket some more, Aurelien and I were confused about the objections to the ticket from the forum at large. It appeared that some of the objections reported by Dave on the ticket might have come from a misunderstanding in the forum of what the ticket meant. The proposed plan at this point is to discuss the ticket during our plenary in San Jose to try to discern the objection so we can bring a new version of this ticket if necessary or start the process again if the text is good. >> >> On May 7, 2013, at 4:53 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> >>> author: jjhursey >>> >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 >>> >>> This ticket essentially links MPI_ERRORS_ARE_FATAL on a communicator to calling MPI_ABORT on the communicator, i.e. only the processes in that communicator are aborted, while other communicators could potentially remain functional. >>> >>> There was much discussion on the ticket about the scope of this change, and in the end the ticket has remained stagnant for about a year because of it, however I don't think that the changes here should be too controversial. According to the ticket, the main argument against it at the Japan meeting was that for some types of functions, there is not a request which can be used to provide error checking and therefore when an error occurs, the entire application would be forced to fall back to MPI_ERRORS_ARE_FATAL despite setting another error handler, therefore making FT difficult. Some alternate text was provided on the ticket. >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL
by George Bosilca 13 May '13

13 May '13

While I'm not supporting the functionality proposed on this ticket, I don't see how this ticket is a duplicate of MPI_COMM_REVOKE. This function allows any process to propagate the exclusion of any other process from the known universe, based on local information. Such behavior is difficult to implement in ULFM as it will require an agreement, thus a synchronizing operation. The only similarity I see with MPI_COMM_REVOKE is the way the kill message is propagated (operation with a global scope but without global call). George. On May 8, 2013, at 11:27 , Wesley Bland <wbland(a)mcs.anl.gov> wrote: > I plan to post the result of any discussions we have onto the ticket, but I know that most of this list isn't on the ticket so we'd miss out on most of the FT people. > > On May 8, 2013, at 10:23 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > >> It might be helpful to have this discussion on the ticket itself (or copy/paste it there), so that it will be saved for future reference. >> >> ~Jim. >> >> On 5/8/13 10:21 AM, Richard Graham wrote: >>> I believe this received the "cold shoulder", and should be closed. >>> >>> -----Original Message----- >>> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aurï¿½lien Bouteiller >>> Sent: Wednesday, May 08, 2013 11:10 AM >>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >>> Subject: Re: [Mpi3-ft] Ticket #292: MPI_COMM_KILL >>> >>> This is a problem child. It changes the state of valid communicators by making operations operate on sparse communicators that contain holes (failed processes). This is difficult to implement and will have adverse performance hit, even outside failure cases. The functionality is duplicated with revoke/shrink, group_create, etc. I'm for killing this one, it has become irrelevant in the new context. >>> >>> Aurelien >>> >>> Le 7 mai 2013 ï¿½ 17:19, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : >>> >>>> author: jjhursey >>>> >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/292 >>>> >>>> This proposal put forth a new function, MPI_COMM_KILL, that would exclude a remote rank from any further communication in all communicators in the MPI universe. >>>> >>>> One of the concerns of the forum is that it was starting to define semantics for failure scenarios other than fail-stop errors, specifically transient/intermittent failures. This was moved out of the previous RTS proposal because of this distinction. In the context of the current proposal, ULFM, this isn't really required anymore as the ULFM proposal states that once a process starts exhibiting failure, it is treated as fail-stop and should be excluded from further participation. >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> -- >>> * Dr. Aurï¿½lien Bouteiller >>> * Researcher at Innovative Computing Laboratory >>> * University of Tennessee >>> * 1122 Volunteer Boulevard, suite 309b >>> * Knoxville, TN 37996 >>> * 865 974 9375 >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #324: Clarify MPI_ERRORS_ARE_FATAL scope of abort
by George Bosilca 13 May '13

13 May '13

Dave points are entirely valid and represent a subtle [corner-]case in the standard. Orphaned non-completed requests, in the sense that the request was freed (MPI_Request_free) and the associated communicator was freed as well (MPI_Comm_free) are defined as raising errors on MPI_COMM_WORLD. Thus, the scope of the request become global, and a fault on such particular requests will bring down the entire MPI_COMM_WORLD (which is against the original scope of the ticket, at least the first part). George. On May 13, 2013, at 12:05 , Wesley Bland <wbland(a)mcs.anl.gov> wrote: > After looking at this ticket some more, Aurelien and I were confused about the objections to the ticket from the forum at large. It appeared that some of the objections reported by Dave on the ticket might have come from a misunderstanding in the forum of what the ticket meant. The proposed plan at this point is to discuss the ticket during our plenary in San Jose to try to discern the objection so we can bring a new version of this ticket if necessary or start the process again if the text is good. > > On May 7, 2013, at 4:53 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> author: jjhursey >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 >> >> This ticket essentially links MPI_ERRORS_ARE_FATAL on a communicator to calling MPI_ABORT on the communicator, i.e. only the processes in that communicator are aborted, while other communicators could potentially remain functional. >> >> There was much discussion on the ticket about the scope of this change, and in the end the ticket has remained stagnant for about a year because of it, however I don't think that the changes here should be too controversial. According to the ticket, the main argument against it at the Japan meeting was that for some types of functions, there is not a request which can be used to provide error checking and therefore when an error occurs, the entire application would be forced to fall back to MPI_ERRORS_ARE_FATAL despite setting another error handler, therefore making FT difficult. Some alternate text was provided on the ticket. > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Ticket #306: Suggest a Return Code Value from mpiexec
by Wesley Bland 13 May '13

13 May '13

After some discussion it was suggested that this ticket is too complicated to be accepted and would need some major reworking to be brought back. As it's not a priority, it will probably just be withdrawn. Wesley On May 7, 2013, at 4:35 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > author: jjhursey > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/306 > > This ticket is an effort to specify the value of the return code for mpiexec. > > It seems like a reasonable ticket to me and there is no discussion on the ticket page. Does anyone know if this was ever discussed at the forum and if there was any feedback?

1 0

Re: [Mpi3-ft] Ticket #277 - Add MPI_Timer Requests
by Wesley Bland 13 May '13

13 May '13

This ticket doesn't seem to have any defenders at the moment. I'll post our plans on the ticket itself and if nothing comes up by the end of the month, it will be withdrawn. Wesley On May 7, 2013, at 4:04 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > As we discussed on the call last week, there are quite a few tickets on the FTWG list, many of which have been largely untouched for at least a year, and many times, much longer. I'm going to try to start a thread for each of the tickets to get a conversation going about whether or not we should keep these tickets around or clean them up and mark them as retracted. I'll try to give a brief summary of the ticket, though I encourage everyone to take a look at the ticket to have a chance to read any notes made there. > > To establish some ground rules, I don't propose removing any tickets just because the person who started them doesn't read their email today. I think we should let these things stick around for a while longer to give people a chance to catch up. I think if we don't hear anything defending a ticket by the next forum meeting (beginning of June), we can consider the ticket as dormant and can withdraw them. I'm certainly open to suggestion otherwise. > > ---- > > author: ftillier > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/277 > > This ticket discusses a middle ground between MPI_TEST and MPI_WAIT where requests can have a timeout value attached to them which would limit the amount of time spent in an MPI_WAIT operation. Request timers could also be reset using MPI_TIMER_RESET and cancelled using MPI_CANCEL. > > This ticket hasn't been touched since early 2012.

1 0

Re: [Mpi3-ft] Ticket #324: Clarify MPI_ERRORS_ARE_FATAL scope of abort
by Wesley Bland 13 May '13

13 May '13

After looking at this ticket some more, Aurelien and I were confused about the objections to the ticket from the forum at large. It appeared that some of the objections reported by Dave on the ticket might have come from a misunderstanding in the forum of what the ticket meant. The proposed plan at this point is to discuss the ticket during our plenary in San Jose to try to discern the objection so we can bring a new version of this ticket if necessary or start the process again if the text is good. On May 7, 2013, at 4:53 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > author: jjhursey > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 > > This ticket essentially links MPI_ERRORS_ARE_FATAL on a communicator to calling MPI_ABORT on the communicator, i.e. only the processes in that communicator are aborted, while other communicators could potentially remain functional. > > There was much discussion on the ticket about the scope of this change, and in the end the ticket has remained stagnant for about a year because of it, however I don't think that the changes here should be too controversial. According to the ticket, the main argument against it at the Japan meeting was that for some types of functions, there is not a request which can be used to provide error checking and therefore when an error occurs, the entire application would be forced to fall back to MPI_ERRORS_ARE_FATAL despite setting another error handler, therefore making FT difficult. Some alternate text was provided on the ticket.

1 0

Re: [Mpi3-ft] Ticket #307: Behavior of MPI_INIT after failure to initialize
by Wesley Bland 10 May '13

10 May '13

After some offline discussion, Aurelien and I think that we should probably keep the text as is. It will be more backward compatible and provides the best guidance for MPI_INIT behavior. This ticket can probably go. On May 8, 2013, at 10:23 AM, Richard Graham <richardg(a)mellanox.com> wrote: > I do not thing there was any conclusion here - this is even true when one is not worrying about FT. What do you do when mpirun does not give you the number of process you asked for ? Before MPI_INIT we are not guaranteed to have more than a single "process". > > Rich > > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Wesley Bland > Sent: Wednesday, May 08, 2013 11:18 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] Ticket #307: Behavior of MPI_INIT after failure to initialize > > Is there anything else from this ticket that we would find useful before killing it? For example, all of the text we have about MPI_INIT in ULFM is in an advice to implementors. Should we move anything to normative text? > > On May 8, 2013, at 10:06 AM, Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> wrote: > >> My memory is that this ticket has been combined, upgraded and included into ULFM. >> >> >> Le 7 mai 2013 ï¿½ 17:40, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿½crit : >> >>> author: jjhursey >>> >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/307 >>> >>> This ticket specifies the behavior of the MPI library after a during MPI_INIT. Basically, if there is a failure during INIT, the library should return an appropriate return code and calling any other MPI function (with a few exceptions) is erroneous and will trigger the error handler for MPI_COMM_WORLD, which by default is MPI_ERRORS_ARE_FATAL and cannot be changed in this instance. >>> >>> Again, this ticket has no discussion attached to it. I remember some discussions from the RTS proposal days, but I don't remember the results of any of those discussions. Can anyone else bring this back to light? >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> -- >> * Dr. Aurï¿½lien Bouteiller >> * Researcher at Innovative Computing Laboratory >> * University of Tennessee >> * 1122 Volunteer Boulevard, suite 309b >> * Knoxville, TN 37996 >> * 865 974 9375 >> >> >> >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0