November 2013 - mpiwg-ft - mailman3.cels.anl.gov

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by George Bosilca 27 Nov '13

27 Nov '13

> > On Nov 27, 2013, at 20:54 , Richard Graham <richardg(a)mellanox.com> wrote: > >> On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> wrote: >> >> >> I am thinking about the next step, and have some questions on the semantics of MPI_Comm_revoke() >> >> What next step are you referring to? > [rich] To the full recovery stage. Post what we are talking about now. Full recovery stage? Can you expose a little more details here please. >> >> >> - When the routine returns, can the communicator ever be used again ? If I remember correctly, the communicator is available for point-to-point traffic, but not collective traffic – is this correct ? >> >> A revoked communicator is unable to support any communication (point-to-point or collective) with the exception of agree and shrink. If this is not clear enough in the current version of the proposal we should definitively address it. > [rich] does this mean all current state (aside from who is alive) associated with the communicator is gone ? Every deterministic information is still available (info and attributes). You can look for the group of processes associated with the communicator, as well as the group of failed. If what you are looking for is the possible unexpected messages, this is up to the implementation (see below). > Can’t rely on continuing sending pending messages ? Not on a revoked communicator. If continuing to exchange messages is a requirement, the communicator should not be revoked. >> >> Looking forward, if one wants to restart the failed ranks (let’s assume we add support for this), what can be assume about the “repaired” communicator ? What can’t I assume about this communicator ? >> >> What you can assume depends on what is the meaning of “repaired”. Already today one can spawn new processes and reconstruct a communicator identical to the original communicator before any fault. This can be done using MPI dynamics together with the agreement available in the ULFM proposal. > [rich] This implies that all outstanding traffic is flushed – is this correct ? This is up to the MPI implementation. This is specified on the first “Advice to implementors” on the second page. George.

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by Richard Graham 27 Nov '13

27 Nov '13

PSB From: mpiwg-ft [mailto:[email protected]] On Behalf Of George Bosilca Sent: Wednesday, November 27, 2013 2:48 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] MPI_Comm_revoke behavior On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com<mailto:[email protected]>> wrote: I am thinking about the next step, and have some questions on the semantics of MPI_Comm_revoke() What next step are you referring to? [rich] To the full recovery stage. Post what we are talking about now. - When the routine returns, can the communicator ever be used again ? If I remember correctly, the communicator is available for point-to-point traffic, but not collective traffic - is this correct ? A revoked communicator is unable to support any communication (point-to-point or collective) with the exception of agree and shrink. If this is not clear enough in the current version of the proposal we should definitively address it. [rich] does this mean all current state (aside from who is alive) associated with the communicator is gone ? Can't rely on continuing sending pending messages ? Looking forward, if one wants to restart the failed ranks (let's assume we add support for this), what can be assume about the "repaired" communicator ? What can't I assume about this communicator ? What you can assume depends on what is the meaning of "repaired". Already today one can spawn new processes and reconstruct a communicator identical to the original communicator before any fault. This can be done using MPI dynamics together with the agreement available in the ULFM proposal. [rich] This implies that all outstanding traffic is flushed - is this correct ? George. Rich _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] MPI_Comm_revoke behavior
by George Bosilca 27 Nov '13

27 Nov '13

On Nov 27, 2013, at 20:33 , Richard Graham <richardg(a)mellanox.com> wrote: > I am thinking about the next step, and have some questions on the semantics of MPI_Comm_revoke() What next step are you referring to? > - When the routine returns, can the communicator ever be used again ? If I remember correctly, the communicator is available for point-to-point traffic, but not collective traffic – is this correct ? A revoked communicator is unable to support any communication (point-to-point or collective) with the exception of agree and shrink. If this is not clear enough in the current version of the proposal we should definitively address it. > - Looking forward, if one wants to restart the failed ranks (let’s assume we add support for this), what can be assume about the “repaired” communicator ? What can’t I assume about this communicator ? What you can assume depends on what is the meaning of “repaired”. Already today one can spawn new processes and reconstruct a communicator identical to the original communicator before any fault. This can be done using MPI dynamics together with the agreement available in the ULFM proposal. George. > Rich > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

MPI_Comm_revoke behavior
by Richard Graham 27 Nov '13

27 Nov '13

I am thinking about the next step, and have some questions on the semantics of MPI_Comm_revoke() - When the routine returns, can the communicator ever be used again ? If I remember correctly, the communicator is available for point-to-point traffic, but not collective traffic - is this correct ? - Looking forward, if one wants to restart the failed ranks (let's assume we add support for this), what can be assume about the "repaired" communicator ? What can't I assume about this communicator ? Rich

1 0

MPI FTWG Con Call 2013-11-26
by Wesley Bland 26 Nov '13

26 Nov '13

Dear WG members, This is a reminder that according to our planning, we are having our regular phone meeting TODAY at 3pm EDT. I won’t be able to attend the call this week, but there probably needs to be some discussion about the next forum meeting. Now that we aren’t going to be doing a reading, we won’t need to prepare for that just yet, but we will still be presenting a plenary that will be similar to the time we had in Madrid. I’ve attached the slides that Rich presented here as a starting point. Please take a look and send back any notes that you may have. Date: November 26 Time: 3pm EDT/New York Dial-in information: 712-432-0360 Code: 623998# Agenda: * Plans for December Forum Meeting

1 0

Re: [mpiwg-ft] A meeting this week
by Jim Dinan 22 Nov '13

22 Nov '13

Got it, thanks guys. ~Jim. On Fri, Nov 22, 2013 at 3:15 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > No, it's a correctness thing. You're only required to detect failures for > processes you're actively communicating with (in a collective, > receiving/sending a message). If you're not directly communicating, the > implementation isn't required to notify you of a failure (and it would be > confusing if it did). In this case, you might need the revoke to notify the > other processes manually an example might be a stencil computation where > nor process detects a failure and decides to branch to a recovery path. You > might need to revoke the communicator first before branching. > > Wesley > > On Nov 22, 2013, at 2:02 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > So, what is the argument for having MPI_Comm/win_revoke? Is it a > performance, rather than a correctness argument? > > ~Jim. > > > On Fri, Nov 22, 2013 at 2:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> It doesn't require an asynchronous failure detector. It does require that >> you detect failures (in a unspecified way) insofar as it prevents >> completion. Once you enter the MPI library, you have to use some sort of >> detector (probably from the runtime level) to keep from getting deadlocked. >> >> Wesley >> >> On Nov 22, 2013, at 12:53 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >> The latter case requires a failure detector, right? I had thought the >> current design would avoid this requirement. >> >> ~Jim. >> >> >> On Thu, Nov 21, 2013 at 12:06 PM, Wesley Bland <wbland(a)mcs.anl.gov>wrote: >> >>> No problem about missing the call yesterday. It was a last minute thing. >>> I think we're in good shape to submit the text on Monday, but we're just >>> doing some final passes over the text. There were a few changes that >>> Aurélien will be making and I'm getting Gail to do an English pass, but >>> overall, it's still essentially the same as what we read last year (per the >>> request from the forum). >>> >>> There are a couple of ways out of this deadlock. The first is, as you >>> mentioned, to have a function in the library to essentially manually >>> trigger an error handler and let the library figure out what is wrong. This >>> method would work, but it is a bit heavy handed. The alternative solution >>> is that the wildcard on process X should return an error because the >>> failure of process Y meets the definition if an "involved process." Process >>> X will get an exception (or an MPI_Errhandler) and can trigger the recovery >>> path. >>> >>> Either way should work, but the later is obviously the preferred and >>> expected solution. >>> >>> Thanks, >>> Wesley >>> >>> On Nov 21, 2013, at 10:58 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>> >>> Hi Guys, >>> >>> Sorry I wasn't able to attend. I'm back from SC now, if you need me. >>> >>> I have a concern about the current approach to revoking communicators. >>> Consider a program that uses a library with a communicator, CL, that is >>> private to the library. Process X makes a call to this library and >>> performs a wildcard receive on CL. Process Y fails; Y would have sent a >>> message to X on CL. Process Z sees that Y failed, but it sees it in the >>> user code, outside of the library. Process Z cannot call revoke on CL >>> because it does not have any knowledge about how the library is implemented >>> and it does not have a handle to CL. >>> >>> This seems like a situation that will result in deadlock, unless the >>> library is also extended to include a "respond to process failure" >>> function. Is this handled in some other way, and I'm just not seeing it? >>> >>> It seems like the revoke(comm) approach requires the programmer to know >>> about all communication and all communicators/windows in use in their >>> entire application, including those contained within libraries. Is that a >>> correct assessment? >>> >>> ~Jim. >>> >>> >>> On Wed, Nov 20, 2013 at 2:39 PM, Aurélien Bouteiller < >>> bouteill(a)icl.utk.edu> wrote: >>> >>>> Rich, this is a followup of the proofreading work done during the >>>> regular meeting we had last week, and everybody, including SC attendees, >>>> had a chance to join. I am sorry you couldn’t. >>>> >>>> Anyway, here is the working document for today: all diffs since the >>>> introduction of the new RMA chapter 5 month ago. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Le 19 nov. 2013 à 17:07, Richard Graham <richardg(a)mellanox.com> a >>>> écrit : >>>> >>>> > With SC this week this is poor timing >>>> > >>>> > Rich >>>> > >>>> > ------Original Message------ >>>> > From: Wesley Bland >>>> > To: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>> > Cc: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>> > ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working >>>> Group >>>> > Subject: Re: [mpiwg-ft] A meeting this week >>>> > Sent: Nov 19, 2013 2:13 PM >>>> > >>>> > Ok. I'll be there. I'll send it off for an editing today. >>>> > >>>> > Wesley >>>> > >>>> >> On Nov 19, 2013, at 3:12 PM, Aurélien Bouteiller < >>>> bouteill(a)icl.utk.edu> wrote: >>>> >> >>>> >> Dear WG members, >>>> >> >>>> >> We have been misreading the new forum rules. We have to buckle the >>>> text of the proposal this week and not in 2 weeks from now, so time is >>>> running short. I would like to invite you to a supplementary meeting >>>> tomorrow to make a review of the text together. >>>> >> >>>> >> Jim, I don’t know if you will be able to attend on short notice, but >>>> your input would be greatly appreciated. >>>> >> >>>> >> Date: Wed, November 20, >>>> >> Time: 3pm EDT/New York >>>> >> Dial-in information: 712-432-0360 >>>> >> Code: 623998# >>>> >> >>>> >> Agenda: >>>> >> Review of ULFM text and final work. >>>> >> >>>> >> Aurelien >>>> >> >>>> >> -- >>>> >> * Dr. Aurélien Bouteiller >>>> >> * Researcher at Innovative Computing Laboratory >>>> >> * University of Tennessee >>>> >> * 1122 Volunteer Boulevard, suite 309b >>>> >> * Knoxville, TN 37996 >>>> >> * 865 974 9375 >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> mpiwg-ft mailing list >>>> >> mpiwg-ft(a)lists.mpi-forum.org >>>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> > _______________________________________________ >>>> > mpiwg-ft mailing list >>>> > mpiwg-ft(a)lists.mpi-forum.org >>>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> > _______________________________________________ >>>> > mpiwg-ft mailing list >>>> > mpiwg-ft(a)lists.mpi-forum.org >>>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> -- >>>> * Dr. Aurélien Bouteiller >>>> * Researcher at Innovative Computing Laboratory >>>> * University of Tennessee >>>> * 1122 Volunteer Boulevard, suite 309b >>>> * Knoxville, TN 37996 >>>> * 865 974 9375 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

Re: [mpiwg-ft] A meeting this week
by Wesley Bland 22 Nov '13

22 Nov '13

No, it's a correctness thing. You're only required to detect failures for processes you're actively communicating with (in a collective, receiving/sending a message). If you're not directly communicating, the implementation isn't required to notify you of a failure (and it would be confusing if it did). In this case, you might need the revoke to notify the other processes manually an example might be a stencil computation where nor process detects a failure and decides to branch to a recovery path. You might need to revoke the communicator first before branching. Wesley > On Nov 22, 2013, at 2:02 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > So, what is the argument for having MPI_Comm/win_revoke? Is it a performance, rather than a correctness argument? > > ~Jim. > > >> On Fri, Nov 22, 2013 at 2:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> It doesn't require an asynchronous failure detector. It does require that you detect failures (in a unspecified way) insofar as it prevents completion. Once you enter the MPI library, you have to use some sort of detector (probably from the runtime level) to keep from getting deadlocked. >> >> Wesley >> >>> On Nov 22, 2013, at 12:53 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>> >>> The latter case requires a failure detector, right? I had thought the current design would avoid this requirement. >>> >>> ~Jim. >>> >>> >>>> On Thu, Nov 21, 2013 at 12:06 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >>>> No problem about missing the call yesterday. It was a last minute thing. I think we're in good shape to submit the text on Monday, but we're just doing some final passes over the text. There were a few changes that Aurélien will be making and I'm getting Gail to do an English pass, but overall, it's still essentially the same as what we read last year (per the request from the forum). >>>> >>>> There are a couple of ways out of this deadlock. The first is, as you mentioned, to have a function in the library to essentially manually trigger an error handler and let the library figure out what is wrong. This method would work, but it is a bit heavy handed. The alternative solution is that the wildcard on process X should return an error because the failure of process Y meets the definition if an "involved process." Process X will get an exception (or an MPI_Errhandler) and can trigger the recovery path. >>>> >>>> Either way should work, but the later is obviously the preferred and expected solution. >>>> >>>> Thanks, >>>> Wesley >>>> >>>>> On Nov 21, 2013, at 10:58 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>> >>>>> Hi Guys, >>>>> >>>>> Sorry I wasn't able to attend. I'm back from SC now, if you need me. >>>>> >>>>> I have a concern about the current approach to revoking communicators. Consider a program that uses a library with a communicator, CL, that is private to the library. Process X makes a call to this library and performs a wildcard receive on CL. Process Y fails; Y would have sent a message to X on CL. Process Z sees that Y failed, but it sees it in the user code, outside of the library. Process Z cannot call revoke on CL because it does not have any knowledge about how the library is implemented and it does not have a handle to CL. >>>>> >>>>> This seems like a situation that will result in deadlock, unless the library is also extended to include a "respond to process failure" function. Is this handled in some other way, and I'm just not seeing it? >>>>> >>>>> It seems like the revoke(comm) approach requires the programmer to know about all communication and all communicators/windows in use in their entire application, including those contained within libraries. Is that a correct assessment? >>>>> >>>>> ~Jim. >>>>> >>>>> >>>>>> On Wed, Nov 20, 2013 at 2:39 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>>>>> Rich, this is a followup of the proofreading work done during the regular meeting we had last week, and everybody, including SC attendees, had a chance to join. I am sorry you couldn’t. >>>>>> >>>>>> Anyway, here is the working document for today: all diffs since the introduction of the new RMA chapter 5 month ago. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Le 19 nov. 2013 à 17:07, Richard Graham <richardg(a)mellanox.com> a écrit : >>>>>> >>>>>> > With SC this week this is poor timing >>>>>> > >>>>>> > Rich >>>>>> > >>>>>> > ------Original Message------ >>>>>> > From: Wesley Bland >>>>>> > To: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>>>> > Cc: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>>>> > ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>>>> > Subject: Re: [mpiwg-ft] A meeting this week >>>>>> > Sent: Nov 19, 2013 2:13 PM >>>>>> > >>>>>> > Ok. I'll be there. I'll send it off for an editing today. >>>>>> > >>>>>> > Wesley >>>>>> > >>>>>> >> On Nov 19, 2013, at 3:12 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>>>>> >> >>>>>> >> Dear WG members, >>>>>> >> >>>>>> >> We have been misreading the new forum rules. We have to buckle the text of the proposal this week and not in 2 weeks from now, so time is running short. I would like to invite you to a supplementary meeting tomorrow to make a review of the text together. >>>>>> >> >>>>>> >> Jim, I don’t know if you will be able to attend on short notice, but your input would be greatly appreciated. >>>>>> >> >>>>>> >> Date: Wed, November 20, >>>>>> >> Time: 3pm EDT/New York >>>>>> >> Dial-in information: 712-432-0360 >>>>>> >> Code: 623998# >>>>>> >> >>>>>> >> Agenda: >>>>>> >> Review of ULFM text and final work. >>>>>> >> >>>>>> >> Aurelien >>>>>> >> >>>>>> >> -- >>>>>> >> * Dr. Aurélien Bouteiller >>>>>> >> * Researcher at Innovative Computing Laboratory >>>>>> >> * University of Tennessee >>>>>> >> * 1122 Volunteer Boulevard, suite 309b >>>>>> >> * Knoxville, TN 37996 >>>>>> >> * 865 974 9375 >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> _______________________________________________ >>>>>> >> mpiwg-ft mailing list >>>>>> >> mpiwg-ft(a)lists.mpi-forum.org >>>>>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>>>> > _______________________________________________ >>>>>> > mpiwg-ft mailing list >>>>>> > mpiwg-ft(a)lists.mpi-forum.org >>>>>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>>>> > _______________________________________________ >>>>>> > mpiwg-ft mailing list >>>>>> > mpiwg-ft(a)lists.mpi-forum.org >>>>>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>>>> >>>>>> -- >>>>>> * Dr. Aurélien Bouteiller >>>>>> * Researcher at Innovative Computing Laboratory >>>>>> * University of Tennessee >>>>>> * 1122 Volunteer Boulevard, suite 309b >>>>>> * Knoxville, TN 37996 >>>>>> * 865 974 9375 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-ft mailing list >>>>>> mpiwg-ft(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>>> >>>>> _______________________________________________ >>>>> mpiwg-ft mailing list >>>>> mpiwg-ft(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] A meeting this week
by Aurélien Bouteiller 22 Nov '13

22 Nov '13

When doing MPI_Recv(…, rank=2, …),only failures of rank 2 have to be reported. Other failures do not have to be monitored at all (good: on most networks these days, the HPC fabric will report failure of 2 w/o the need for any supplementary actions on the implementation’s part). Now, consider the following example to understand why revoke is necessary (for correctness). In the following example, only the 2 processes neighboring “fail rank” have an opportunity to detect the failure. When they detect the failure, they “break” from the normal loop of the algorithm (presumably to go in Shrink or do some recovery activity). Since they do not want to continue the sendrecv loop, they have to revoke, so that other process do not stall waiting for them to match the sendrecv call. if( rank == failrank ) { if( args.verbose ) printf( "Rank %04d: SEPPUKU\n", rank ); raise( SIGKILL ); } do { rc = MPI_Sendrecv( b1, count, MPI_DOUBLE, (rank+1)>=np? 0:rank+1, 1, b2, count, MPI_DOUBLE, (rank-1)<0? np-1:rank-1, 1, world, MPI_STATUS_IGNORE ); if( MPI_SUCCESS != rc ) { MPI_Error_string( rc, errorstr, &elen); if( MPI_ERR_PROC_FAILED == rc ) { if( args.verbose ) printf( "Rank %04d: exception (%s); calling Comm_revoke\n", rank, errorstr ); OMPI_Comm_revoke( world ); break; } else if( MPI_ERR_REVOKED == rc ) { break; } } } while( 1 ); I hope this clarifies. Le 22 nov. 2013 à 15:02, Jim Dinan <james.dinan(a)gmail.com> a écrit : > So, what is the argument for having MPI_Comm/win_revoke? Is it a performance, rather than a correctness argument? > > ~Jim. > > > On Fri, Nov 22, 2013 at 2:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > It doesn't require an asynchronous failure detector. It does require that you detect failures (in a unspecified way) insofar as it prevents completion. Once you enter the MPI library, you have to use some sort of detector (probably from the runtime level) to keep from getting deadlocked. > > Wesley > > On Nov 22, 2013, at 12:53 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> The latter case requires a failure detector, right? I had thought the current design would avoid this requirement. >> >> ~Jim. >> >> >> On Thu, Nov 21, 2013 at 12:06 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> No problem about missing the call yesterday. It was a last minute thing. I think we're in good shape to submit the text on Monday, but we're just doing some final passes over the text. There were a few changes that Aurélien will be making and I'm getting Gail to do an English pass, but overall, it's still essentially the same as what we read last year (per the request from the forum). >> >> There are a couple of ways out of this deadlock. The first is, as you mentioned, to have a function in the library to essentially manually trigger an error handler and let the library figure out what is wrong. This method would work, but it is a bit heavy handed. The alternative solution is that the wildcard on process X should return an error because the failure of process Y meets the definition if an "involved process." Process X will get an exception (or an MPI_Errhandler) and can trigger the recovery path. >> >> Either way should work, but the later is obviously the preferred and expected solution. >> >> Thanks, >> Wesley >> >> On Nov 21, 2013, at 10:58 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >>> Hi Guys, >>> >>> Sorry I wasn't able to attend. I'm back from SC now, if you need me. >>> >>> I have a concern about the current approach to revoking communicators. Consider a program that uses a library with a communicator, CL, that is private to the library. Process X makes a call to this library and performs a wildcard receive on CL. Process Y fails; Y would have sent a message to X on CL. Process Z sees that Y failed, but it sees it in the user code, outside of the library. Process Z cannot call revoke on CL because it does not have any knowledge about how the library is implemented and it does not have a handle to CL. >>> >>> This seems like a situation that will result in deadlock, unless the library is also extended to include a "respond to process failure" function. Is this handled in some other way, and I'm just not seeing it? >>> >>> It seems like the revoke(comm) approach requires the programmer to know about all communication and all communicators/windows in use in their entire application, including those contained within libraries. Is that a correct assessment? >>> >>> ~Jim. >>> >>> >>> On Wed, Nov 20, 2013 at 2:39 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>> Rich, this is a followup of the proofreading work done during the regular meeting we had last week, and everybody, including SC attendees, had a chance to join. I am sorry you couldn’t. >>> >>> Anyway, here is the working document for today: all diffs since the introduction of the new RMA chapter 5 month ago. >>> >>> >>> >>> >>> >>> >>> >>> >>> Le 19 nov. 2013 à 17:07, Richard Graham <richardg(a)mellanox.com> a écrit : >>> >>> > With SC this week this is poor timing >>> > >>> > Rich >>> > >>> > ------Original Message------ >>> > From: Wesley Bland >>> > To: MPI WG Fault Tolerance and Dynamic Process Control working Group >>> > Cc: MPI WG Fault Tolerance and Dynamic Process Control working Group >>> > ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working Group >>> > Subject: Re: [mpiwg-ft] A meeting this week >>> > Sent: Nov 19, 2013 2:13 PM >>> > >>> > Ok. I'll be there. I'll send it off for an editing today. >>> > >>> > Wesley >>> > >>> >> On Nov 19, 2013, at 3:12 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>> >> >>> >> Dear WG members, >>> >> >>> >> We have been misreading the new forum rules. We have to buckle the text of the proposal this week and not in 2 weeks from now, so time is running short. I would like to invite you to a supplementary meeting tomorrow to make a review of the text together. >>> >> >>> >> Jim, I don’t know if you will be able to attend on short notice, but your input would be greatly appreciated. >>> >> >>> >> Date: Wed, November 20, >>> >> Time: 3pm EDT/New York >>> >> Dial-in information: 712-432-0360 >>> >> Code: 623998# >>> >> >>> >> Agenda: >>> >> Review of ULFM text and final work. >>> >> >>> >> Aurelien >>> >> >>> >> -- >>> >> * Dr. Aurélien Bouteiller >>> >> * Researcher at Innovative Computing Laboratory >>> >> * University of Tennessee >>> >> * 1122 Volunteer Boulevard, suite 309b >>> >> * Knoxville, TN 37996 >>> >> * 865 974 9375 >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> mpiwg-ft mailing list >>> >> mpiwg-ft(a)lists.mpi-forum.org >>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> > _______________________________________________ >>> > mpiwg-ft mailing list >>> > mpiwg-ft(a)lists.mpi-forum.org >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> > _______________________________________________ >>> > mpiwg-ft mailing list >>> > mpiwg-ft(a)lists.mpi-forum.org >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> -- >>> * Dr. Aurélien Bouteiller >>> * Researcher at Innovative Computing Laboratory >>> * University of Tennessee >>> * 1122 Volunteer Boulevard, suite 309b >>> * Knoxville, TN 37996 >>> * 865 974 9375 >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] A meeting this week
by Jim Dinan 22 Nov '13

22 Nov '13

So, what is the argument for having MPI_Comm/win_revoke? Is it a performance, rather than a correctness argument? ~Jim. On Fri, Nov 22, 2013 at 2:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > It doesn't require an asynchronous failure detector. It does require that > you detect failures (in a unspecified way) insofar as it prevents > completion. Once you enter the MPI library, you have to use some sort of > detector (probably from the runtime level) to keep from getting deadlocked. > > Wesley > > On Nov 22, 2013, at 12:53 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > The latter case requires a failure detector, right? I had thought the > current design would avoid this requirement. > > ~Jim. > > > On Thu, Nov 21, 2013 at 12:06 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> No problem about missing the call yesterday. It was a last minute thing. >> I think we're in good shape to submit the text on Monday, but we're just >> doing some final passes over the text. There were a few changes that >> Aurélien will be making and I'm getting Gail to do an English pass, but >> overall, it's still essentially the same as what we read last year (per the >> request from the forum). >> >> There are a couple of ways out of this deadlock. The first is, as you >> mentioned, to have a function in the library to essentially manually >> trigger an error handler and let the library figure out what is wrong. This >> method would work, but it is a bit heavy handed. The alternative solution >> is that the wildcard on process X should return an error because the >> failure of process Y meets the definition if an "involved process." Process >> X will get an exception (or an MPI_Errhandler) and can trigger the recovery >> path. >> >> Either way should work, but the later is obviously the preferred and >> expected solution. >> >> Thanks, >> Wesley >> >> On Nov 21, 2013, at 10:58 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >> Hi Guys, >> >> Sorry I wasn't able to attend. I'm back from SC now, if you need me. >> >> I have a concern about the current approach to revoking communicators. >> Consider a program that uses a library with a communicator, CL, that is >> private to the library. Process X makes a call to this library and >> performs a wildcard receive on CL. Process Y fails; Y would have sent a >> message to X on CL. Process Z sees that Y failed, but it sees it in the >> user code, outside of the library. Process Z cannot call revoke on CL >> because it does not have any knowledge about how the library is implemented >> and it does not have a handle to CL. >> >> This seems like a situation that will result in deadlock, unless the >> library is also extended to include a "respond to process failure" >> function. Is this handled in some other way, and I'm just not seeing it? >> >> It seems like the revoke(comm) approach requires the programmer to know >> about all communication and all communicators/windows in use in their >> entire application, including those contained within libraries. Is that a >> correct assessment? >> >> ~Jim. >> >> >> On Wed, Nov 20, 2013 at 2:39 PM, Aurélien Bouteiller < >> bouteill(a)icl.utk.edu> wrote: >> >>> Rich, this is a followup of the proofreading work done during the >>> regular meeting we had last week, and everybody, including SC attendees, >>> had a chance to join. I am sorry you couldn’t. >>> >>> Anyway, here is the working document for today: all diffs since the >>> introduction of the new RMA chapter 5 month ago. >>> >>> >>> >>> >>> >>> >>> >>> >>> Le 19 nov. 2013 à 17:07, Richard Graham <richardg(a)mellanox.com> a écrit >>> : >>> >>> > With SC this week this is poor timing >>> > >>> > Rich >>> > >>> > ------Original Message------ >>> > From: Wesley Bland >>> > To: MPI WG Fault Tolerance and Dynamic Process Control working Group >>> > Cc: MPI WG Fault Tolerance and Dynamic Process Control working Group >>> > ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working >>> Group >>> > Subject: Re: [mpiwg-ft] A meeting this week >>> > Sent: Nov 19, 2013 2:13 PM >>> > >>> > Ok. I'll be there. I'll send it off for an editing today. >>> > >>> > Wesley >>> > >>> >> On Nov 19, 2013, at 3:12 PM, Aurélien Bouteiller < >>> bouteill(a)icl.utk.edu> wrote: >>> >> >>> >> Dear WG members, >>> >> >>> >> We have been misreading the new forum rules. We have to buckle the >>> text of the proposal this week and not in 2 weeks from now, so time is >>> running short. I would like to invite you to a supplementary meeting >>> tomorrow to make a review of the text together. >>> >> >>> >> Jim, I don’t know if you will be able to attend on short notice, but >>> your input would be greatly appreciated. >>> >> >>> >> Date: Wed, November 20, >>> >> Time: 3pm EDT/New York >>> >> Dial-in information: 712-432-0360 >>> >> Code: 623998# >>> >> >>> >> Agenda: >>> >> Review of ULFM text and final work. >>> >> >>> >> Aurelien >>> >> >>> >> -- >>> >> * Dr. Aurélien Bouteiller >>> >> * Researcher at Innovative Computing Laboratory >>> >> * University of Tennessee >>> >> * 1122 Volunteer Boulevard, suite 309b >>> >> * Knoxville, TN 37996 >>> >> * 865 974 9375 >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> mpiwg-ft mailing list >>> >> mpiwg-ft(a)lists.mpi-forum.org >>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> > _______________________________________________ >>> > mpiwg-ft mailing list >>> > mpiwg-ft(a)lists.mpi-forum.org >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> > _______________________________________________ >>> > mpiwg-ft mailing list >>> > mpiwg-ft(a)lists.mpi-forum.org >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> -- >>> * Dr. Aurélien Bouteiller >>> * Researcher at Innovative Computing Laboratory >>> * University of Tennessee >>> * 1122 Volunteer Boulevard, suite 309b >>> * Knoxville, TN 37996 >>> * 865 974 9375 >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

Re: [mpiwg-ft] A meeting this week
by Wesley Bland 22 Nov '13

22 Nov '13

It doesn't require an asynchronous failure detector. It does require that you detect failures (in a unspecified way) insofar as it prevents completion. Once you enter the MPI library, you have to use some sort of detector (probably from the runtime level) to keep from getting deadlocked. Wesley > On Nov 22, 2013, at 12:53 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > The latter case requires a failure detector, right? I had thought the current design would avoid this requirement. > > ~Jim. > > >> On Thu, Nov 21, 2013 at 12:06 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> No problem about missing the call yesterday. It was a last minute thing. I think we're in good shape to submit the text on Monday, but we're just doing some final passes over the text. There were a few changes that Aurélien will be making and I'm getting Gail to do an English pass, but overall, it's still essentially the same as what we read last year (per the request from the forum). >> >> There are a couple of ways out of this deadlock. The first is, as you mentioned, to have a function in the library to essentially manually trigger an error handler and let the library figure out what is wrong. This method would work, but it is a bit heavy handed. The alternative solution is that the wildcard on process X should return an error because the failure of process Y meets the definition if an "involved process." Process X will get an exception (or an MPI_Errhandler) and can trigger the recovery path. >> >> Either way should work, but the later is obviously the preferred and expected solution. >> >> Thanks, >> Wesley >> >>> On Nov 21, 2013, at 10:58 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>> >>> Hi Guys, >>> >>> Sorry I wasn't able to attend. I'm back from SC now, if you need me. >>> >>> I have a concern about the current approach to revoking communicators. Consider a program that uses a library with a communicator, CL, that is private to the library. Process X makes a call to this library and performs a wildcard receive on CL. Process Y fails; Y would have sent a message to X on CL. Process Z sees that Y failed, but it sees it in the user code, outside of the library. Process Z cannot call revoke on CL because it does not have any knowledge about how the library is implemented and it does not have a handle to CL. >>> >>> This seems like a situation that will result in deadlock, unless the library is also extended to include a "respond to process failure" function. Is this handled in some other way, and I'm just not seeing it? >>> >>> It seems like the revoke(comm) approach requires the programmer to know about all communication and all communicators/windows in use in their entire application, including those contained within libraries. Is that a correct assessment? >>> >>> ~Jim. >>> >>> >>>> On Wed, Nov 20, 2013 at 2:39 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>>> Rich, this is a followup of the proofreading work done during the regular meeting we had last week, and everybody, including SC attendees, had a chance to join. I am sorry you couldn’t. >>>> >>>> Anyway, here is the working document for today: all diffs since the introduction of the new RMA chapter 5 month ago. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Le 19 nov. 2013 à 17:07, Richard Graham <richardg(a)mellanox.com> a écrit : >>>> >>>> > With SC this week this is poor timing >>>> > >>>> > Rich >>>> > >>>> > ------Original Message------ >>>> > From: Wesley Bland >>>> > To: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>> > Cc: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>> > ReplyTo: MPI WG Fault Tolerance and Dynamic Process Control working Group >>>> > Subject: Re: [mpiwg-ft] A meeting this week >>>> > Sent: Nov 19, 2013 2:13 PM >>>> > >>>> > Ok. I'll be there. I'll send it off for an editing today. >>>> > >>>> > Wesley >>>> > >>>> >> On Nov 19, 2013, at 3:12 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>>> >> >>>> >> Dear WG members, >>>> >> >>>> >> We have been misreading the new forum rules. We have to buckle the text of the proposal this week and not in 2 weeks from now, so time is running short. I would like to invite you to a supplementary meeting tomorrow to make a review of the text together. >>>> >> >>>> >> Jim, I don’t know if you will be able to attend on short notice, but your input would be greatly appreciated. >>>> >> >>>> >> Date: Wed, November 20, >>>> >> Time: 3pm EDT/New York >>>> >> Dial-in information: 712-432-0360 >>>> >> Code: 623998# >>>> >> >>>> >> Agenda: >>>> >> Review of ULFM text and final work. >>>> >> >>>> >> Aurelien >>>> >> >>>> >> -- >>>> >> * Dr. Aurélien Bouteiller >>>> >> * Researcher at Innovative Computing Laboratory >>>> >> * University of Tennessee >>>> >> * 1122 Volunteer Boulevard, suite 309b >>>> >> * Knoxville, TN 37996 >>>> >> * 865 974 9375 >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> mpiwg-ft mailing list >>>> >> mpiwg-ft(a)lists.mpi-forum.org >>>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> > _______________________________________________ >>>> > mpiwg-ft mailing list >>>> > mpiwg-ft(a)lists.mpi-forum.org >>>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> > _______________________________________________ >>>> > mpiwg-ft mailing list >>>> > mpiwg-ft(a)lists.mpi-forum.org >>>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> -- >>>> * Dr. Aurélien Bouteiller >>>> * Researcher at Innovative Computing Laboratory >>>> * University of Tennessee >>>> * 1122 Volunteer Boulevard, suite 309b >>>> * Knoxville, TN 37996 >>>> * 865 974 9375 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0