June 2015 - mpiwg-ft - mailman3.cels.anl.gov

FTWG Con Call
by Aurélien Bouteiller 22 Jun '15

22 Jun '15

Just a quick reminder, we will have our usual FTWG telecom at 3:00 PM Eastern tomorrow. On the agenda: * Continuation of discussion about Tools and Error Handlers * ULFM (#323, #325, #326) * RMA ULFM text changes (if enough interested people show up) Dial-in information: Phone: 712-432-0360 Code: 623998# See you (hear you) tomorrow, Aurelien

1 0

Re: [mpiwg-ft] FTWG Con Call
by Bland, Wesley 09 Jun '15

09 Jun '15

The notes from the call are available on the wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-09 On June 9, 2015 at 7:52:00 AM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: Today is our usual FTWG con call at 3:00 PM Eastern. Dial-in information: Phone: 712-432-0360<tel://Phone:%20712-432-0360> Code: 623998# On the agenda: * Error handlers (#324 & #477) * Tickets: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/477 <http://><https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/477><https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/477> * Slides: https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… * PDF: mpi-report-tickets324-477.pdf<https://svn.mpi-forum.org/trac/mpi-forum-web/raw-attachment/ticket/477/mpi-…> * ULFM (#323, #325, #326) * RMA ULFM text changes Still in progress: * Redesign error handlers for MPI 4 Thanks, Wesley

1 0

FTWG Con Call
by Bland, Wesley 09 Jun '15

09 Jun '15

Today is our usual FTWG con call at 3:00 PM Eastern. Dial-in information: Phone: 712-432-0360<tel://Phone:%20712-432-0360> Code: 623998# On the agenda: * Error handlers (#324 & #477) * Tickets: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/477 <http://><https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/477><https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/477> * Slides: https://docs.google.com/presentation/d/10Sz9aCDezSLH1rss6XYApo_wVqTyHH08G3F… * PDF: mpi-report-tickets324-477.pdf<https://svn.mpi-forum.org/trac/mpi-forum-web/raw-attachment/ticket/477/mpi-…> * ULFM (#323, #325, #326) * RMA ULFM text changes Still in progress: * Redesign error handlers for MPI 4 Thanks, Wesley

1 0

Re: [mpiwg-ft] Ticket 324 June 2015 Reading
by Bland, Wesley 03 Jun '15

03 Jun '15

The new version of the tickets is now available on #324 and #477. We’ll discuss these more in the next con call. You can also check out the updated notes on the wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 On June 2, 2015 at 4:57:33 PM, Fab Tillier (ftillier(a)microsoft.com<mailto:[email protected]>) wrote: Hi Wesley, I think your wording captures exactly what I was thinking. It's up to the application then to ensure it does the right thing for its usage model and error handler constraints. One aspect that will need to be clarified is what the input MPI_Comm handle value will be in an error handler callback if that communicator has been freed. I can see two viable solutions: 1. MPI_COMM_NULL - the user freed the communicator, all attributes have been cleaned up, and no new operations should be initiated on it. 2. a handle valid for the duration of the callback. This allows accessing the communicator, but would require defining a whitelist of API calls. For example, we shouldn't support setting attributes since there won't be a mechanism to free them (the communicator was already freed). In any case, I'm not sure this would be useful, and would want to see a use case to support this. The idea of having a "restricted" handle seems ugly and ripe for us to realize we overlooked a problematic API call that we allowed in the whitelist. Cheers, -Fab -----Original Message----- From: Bland, Wesley [mailto:[email protected]] bbr/>Sent: Wednesday, June 3, 2015 1:03 AM To: MPI WG Fault Tolerance and Dynamic Process Control working Group; Fab Tillier Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading I think the thing that makes the most sense from a user’s perspective is to say that the error handler that was last set on the communicator when it was freed by the user will be called. I haven’t thought through all of the ramifications of how this could cause problems (of which I am sure there are many), but this seems to follow the law of least astonishment. If you set an error handler, you would expect it to be called. On the other hand, I’m all for telling people not to free their communicators until all of the requests attached to them are done, especially if you want proper error handling. Those are the two options that make the most sense to me. On June 2, 2015 at 12:27:13 AM, Fab Tillier (ftillier(a)microsoft.com<mailto:[email protected]>) wrote: Hi Martin, I think it's important to distinguish between communicator handle validity from the underlying communicator object's validity. You can think of MPI_COMM_FREE as freeing the handle and its reference on the underlying object (thus marking the communicator for deletion). Internally, the MPI implementation can keep the object around until all (other) references are freed (e.g. all pending operations complete). In case of an error, the implementation still has a valid object with which to call MPI_Abort, even if the user doesn't have a valid handle to that object (i.e. the user could not have called MPI_Abort themselves). The handle to such a communicator is effectively MPI_COMM_NULL. An application that wants higher fidelity for the handle being reported as the source should not have freed that communicator while operations were pending on it. I don't see why we need to define special behavior so that a user can make an equivalent call to MPI_Abort. -Fab -----Original Message----- From: mpiwg-ft [mailto:[email protected]] On Behalf Of Schulz Martin Sent: Tuesday, June 2, 2015 7:10 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading Hi Fab, all, I see where you are coming from and this makes sense to me. Defining this behavior more accurately and also defining what happens when a fault happens on freed communicator is clearly needed. The current standard text pretty much keeps the current communicator around, even if it is freed, and that's probably the right way to do it. However, the current text still treats it as freed and so we can't easily call Abort on it. So, the question is what to do with the Abort if a fault happens in such a scenario. I think we have four options: abort only MPI_COMM_SELF (this seems too localized to me), abort all processors in the communicator (I don't think we can easily do this, since we don't have the communicator anymore, unless we change the behavior of free), abort all processes in MPI_COMM_WORLD (which seems too draconian), or to abort the immediate parent in the communicator hierarchy (which I am suggesting). An orthogonal issue is what to do, if we change the behavior of the error handler - do we follow the error handler of the actual, but freed communicator, or do we take the different handler of the parent. I don't have a good answer for this. Martin ` ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA On 6/1/15, 11:36 PM, "Fab Tillier" <ftillier(a)microsoft.com> wrote: >I think propagating upwards is weird, and only there to work around >poorly defined semantics at the cost of backward compatibility. > >Take an application that makes a dup of MPI_COMM_WORLD, then changes the >errhandler to MPI_ERRORS_RETURN. If this app then frees the dup >communicator handle after issuing an MPI_Irecv, it would currently likely >expect MPI_ERRORS_RETURN to be invoked if that receive encounters an >error, rather than MPI_ERRORS_ARE_FATAL that exists on MPI_COMM_WORLD. > >If an applications has a custom error handler that has per-communicator >state, it would need to either: >- keep track of outstanding requests on the communicator before freeing >the handle >- change the error handler to one of the built-in ones before freeing >- call MPI_COMM_DISCONNECT to ensure all pending communications are >complete (with potential ramifications for anything < MPI_THREAD_MULTIPLE) > >I think clarifying that an application that requires per-communicator >context in their error handler and calls MPI_COMM_FREE before all pending >operations are complete is erroneous is probably a better scoped solution >- MPI has never provided any mechanism to notify the application of when >a communicator is actually freed, and clearly defines that it may not be >freed when MPI_COMM_FREE returns. > >-Fab > >-----Original Message----- >From: mpiwg-ft [mailto:[email protected]] On Behalf Of >Schulz Martin >Sent: Tuesday, June 2, 2015 5:06 PM >To: MPI WG Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading > >Hi Wesley, > >Thanks for the summary - I think this describes it fairly well. > >One additional comment: there was some discussion on having to deal with >situations where there is/was an associated communicator, but that >communicator has been freed before the fault happened. The question was >which communicator an Abort should be on in this case. Personally, I think >it should propagate upwards to the next communicator in the hierarchy, >worst case to COMM_WORLD, but other options exist as well. It would be >good, though, to clearly define this case. > >Martin > > >________________________________________________________________________ >Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > >On 6/1/15, 10:00 PM, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > >>Notes from the ticket reading are now posted on the wiki: >>https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 >> >>TL;DR - The reading did not ³pass², but we got lots of good feedback to >>come back with a new version. We should consider splitting this into two >>or three tickets. One to define new errhandlers that does the new >>definition (abort communicator) and one that¹s a more well defined old >>definition (abort MPI_COMM_WORLD). Another ticket will deprecate >>MPI_COMM_ERRORS_ARE_FATAL. Another ticket will consolidate the >>definitions of all of the error handling text to a single place. >> >>The rest of the details can be found in the wiki. >> >>I¹ll be working on some drafts over the next few days to try to get new >>versions of this ticket out for discussion. My tentative hope is to get >>this ready for a new plenary in September. There¹s going to be enough >>changes that this should probably get a plenary before another reading. >> >>Comments welcome. >> >>Thanks, >>Wesley >>_______________________________________________ >>mpiwg-ft mailing list >>mpiwg-ft(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Ticket 324 June 2015 Reading
by Bland, Wesley 03 Jun '15

03 Jun '15

Hi Fab, I understand what you’re saying and this is something we discussed somewhat in the room at the time, but I think we have changed things enough now (new version posted on the ticket(s) soon) that you shouldn’t get into a situation where an error handler is called on a communicator that’s been freed. If we do have to get into this situation, I’d vote for #1 since that’s how I/O already does this. Thanks, Wesley On June 2, 2015 at 4:57:33 PM, Fab Tillier (ftillier(a)microsoft.com<mailto:[email protected]>) wrote: Hi Wesley, I think your wording captures exactly what I was thinking. It's up to the application then to ensure it does the right thing for its usage model and error handler constraints. One aspect that will need to be clarified is what the input MPI_Comm handle value will be in an error handler callback if that communicator has been freed. I can see two viable solutions: 1. MPI_COMM_NULL - the user freed the communicator, all attributes have been cleaned up, and no new operations should be initiated on it. 2. a handle valid for the duration of the callback. This allows accessing the communicator, but would require defining a whitelist of API calls. For example, we shouldn't support setting attributes since there won't be a mechanism to free them (the communicator was already freed). In any case, I'm not sure this would be useful, and would want to see a use case to support this. The idea of having a "restricted" handle seems ugly and ripe for us to realize we overlooked a problematic API call that we allowed in the whitelist. Cheers, -Fab -----Original Message----- From: Bland, Wesley [mailto:[email protected]] bbr/>Sent: Wednesday, June 3, 2015 1:03 AM To: MPI WG Fault Tolerance and Dynamic Process Control working Group; Fab Tillier Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading I think the thing that makes the most sense from a user’s perspective is to say that the error handler that was last set on the communicator when it was freed by the user will be called. I haven’t thought through all of the ramifications of how this could cause problems (of which I am sure there are many), but this seems to follow the law of least astonishment. If you set an error handler, you would expect it to be called. On the other hand, I’m all for telling people not to free their communicators until all of the requests attached to them are done, especially if you want proper error handling. Those are the two options that make the most sense to me. On June 2, 2015 at 12:27:13 AM, Fab Tillier (ftillier(a)microsoft.com<mailto:[email protected]>) wrote: Hi Martin, I think it's important to distinguish between communicator handle validity from the underlying communicator object's validity. You can think of MPI_COMM_FREE as freeing the handle and its reference on the underlying object (thus marking the communicator for deletion). Internally, the MPI implementation can keep the object around until all (other) references are freed (e.g. all pending operations complete). In case of an error, the implementation still has a valid object with which to call MPI_Abort, even if the user doesn't have a valid handle to that object (i.e. the user could not have called MPI_Abort themselves). The handle to such a communicator is effectively MPI_COMM_NULL. An application that wants higher fidelity for the handle being reported as the source should not have freed that communicator while operations were pending on it. I don't see why we need to define special behavior so that a user can make an equivalent call to MPI_Abort. -Fab -----Original Message----- From: mpiwg-ft [mailto:[email protected]] On Behalf Of Schulz Martin Sent: Tuesday, June 2, 2015 7:10 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading Hi Fab, all, I see where you are coming from and this makes sense to me. Defining this behavior more accurately and also defining what happens when a fault happens on freed communicator is clearly needed. The current standard text pretty much keeps the current communicator around, even if it is freed, and that's probably the right way to do it. However, the current text still treats it as freed and so we can't easily call Abort on it. So, the question is what to do with the Abort if a fault happens in such a scenario. I think we have four options: abort only MPI_COMM_SELF (this seems too localized to me), abort all processors in the communicator (I don't think we can easily do this, since we don't have the communicator anymore, unless we change the behavior of free), abort all processes in MPI_COMM_WORLD (which seems too draconian), or to abort the immediate parent in the communicator hierarchy (which I am suggesting). An orthogonal issue is what to do, if we change the behavior of the error handler - do we follow the error handler of the actual, but freed communicator, or do we take the different handler of the parent. I don't have a good answer for this. Martin ` ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA On 6/1/15, 11:36 PM, "Fab Tillier" <ftillier(a)microsoft.com> wrote: >I think propagating upwards is weird, and only there to work around >poorly defined semantics at the cost of backward compatibility. > >Take an application that makes a dup of MPI_COMM_WORLD, then changes the >errhandler to MPI_ERRORS_RETURN. If this app then frees the dup >communicator handle after issuing an MPI_Irecv, it would currently likely >expect MPI_ERRORS_RETURN to be invoked if that receive encounters an >error, rather than MPI_ERRORS_ARE_FATAL that exists on MPI_COMM_WORLD. > >If an applications has a custom error handler that has per-communicator >state, it would need to either: >- keep track of outstanding requests on the communicator before freeing >the handle >- change the error handler to one of the built-in ones before freeing >- call MPI_COMM_DISCONNECT to ensure all pending communications are >complete (with potential ramifications for anything < MPI_THREAD_MULTIPLE) > >I think clarifying that an application that requires per-communicator >context in their error handler and calls MPI_COMM_FREE before all pending >operations are complete is erroneous is probably a better scoped solution >- MPI has never provided any mechanism to notify the application of when >a communicator is actually freed, and clearly defines that it may not be >freed when MPI_COMM_FREE returns. > >-Fab > >-----Original Message----- >From: mpiwg-ft [mailto:[email protected]] On Behalf Of >Schulz Martin >Sent: Tuesday, June 2, 2015 5:06 PM >To: MPI WG Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading > >Hi Wesley, > >Thanks for the summary - I think this describes it fairly well. > >One additional comment: there was some discussion on having to deal with >situations where there is/was an associated communicator, but that >communicator has been freed before the fault happened. The question was >which communicator an Abort should be on in this case. Personally, I think >it should propagate upwards to the next communicator in the hierarchy, >worst case to COMM_WORLD, but other options exist as well. It would be >good, though, to clearly define this case. > >Martin > > >________________________________________________________________________ >Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > >On 6/1/15, 10:00 PM, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > >>Notes from the ticket reading are now posted on the wiki: >>https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 >> >>TL;DR - The reading did not ³pass², but we got lots of good feedback to >>come back with a new version. We should consider splitting this into two >>or three tickets. One to define new errhandlers that does the new >>definition (abort communicator) and one that¹s a more well defined old >>definition (abort MPI_COMM_WORLD). Another ticket will deprecate >>MPI_COMM_ERRORS_ARE_FATAL. Another ticket will consolidate the >>definitions of all of the error handling text to a single place. >> >>The rest of the details can be found in the wiki. >> >>I¹ll be working on some drafts over the next few days to try to get new >>versions of this ticket out for discussion. My tentative hope is to get >>this ready for a new plenary in September. There¹s going to be enough >>changes that this should probably get a plenary before another reading. >> >>Comments welcome. >> >>Thanks, >>Wesley >>_______________________________________________ >>mpiwg-ft mailing list >>mpiwg-ft(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Ticket 324 June 2015 Reading
by Fab Tillier 02 Jun '15

02 Jun '15

Hi Wesley, I think your wording captures exactly what I was thinking. It's up to the application then to ensure it does the right thing for its usage model and error handler constraints. One aspect that will need to be clarified is what the input MPI_Comm handle value will be in an error handler callback if that communicator has been freed. I can see two viable solutions: 1. MPI_COMM_NULL - the user freed the communicator, all attributes have been cleaned up, and no new operations should be initiated on it. 2. a handle valid for the duration of the callback. This allows accessing the communicator, but would require defining a whitelist of API calls. For example, we shouldn't support setting attributes since there won't be a mechanism to free them (the communicator was already freed). In any case, I'm not sure this would be useful, and would want to see a use case to support this. The idea of having a "restricted" handle seems ugly and ripe for us to realize we overlooked a problematic API call that we allowed in the whitelist. Cheers, -Fab -----Original Message----- From: Bland, Wesley [mailto:[email protected]] Sent: Wednesday, June 3, 2015 1:03 AM To: MPI WG Fault Tolerance and Dynamic Process Control working Group; Fab Tillier Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading I think the thing that makes the most sense from a user’s perspective is to say that the error handler that was last set on the communicator when it was freed by the user will be called. I haven’t thought through all of the ramifications of how this could cause problems (of which I am sure there are many), but this seems to follow the law of least astonishment. If you set an error handler, you would expect it to be called. On the other hand, I’m all for telling people not to free their communicators until all of the requests attached to them are done, especially if you want proper error handling. Those are the two options that make the most sense to me. On June 2, 2015 at 12:27:13 AM, Fab Tillier (ftillier(a)microsoft.com<mailto:[email protected]>) wrote: Hi Martin, I think it's important to distinguish between communicator handle validity from the underlying communicator object's validity. You can think of MPI_COMM_FREE as freeing the handle and its reference on the underlying object (thus marking the communicator for deletion). Internally, the MPI implementation can keep the object around until all (other) references are freed (e.g. all pending operations complete). In case of an error, the implementation still has a valid object with which to call MPI_Abort, even if the user doesn't have a valid handle to that object (i.e. the user could not have called MPI_Abort themselves). The handle to such a communicator is effectively MPI_COMM_NULL. An application that wants higher fidelity for the handle being reported as the source should not have freed that communicator while operations were pending on it. I don't see why we need to define special behavior so that a user can make an equivalent call to MPI_Abort. -Fab -----Original Message----- From: mpiwg-ft [mailto:[email protected]] On Behalf Of Schulz Martin Sent: Tuesday, June 2, 2015 7:10 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading Hi Fab, all, I see where you are coming from and this makes sense to me. Defining this behavior more accurately and also defining what happens when a fault happens on freed communicator is clearly needed. The current standard text pretty much keeps the current communicator around, even if it is freed, and that's probably the right way to do it. However, the current text still treats it as freed and so we can't easily call Abort on it. So, the question is what to do with the Abort if a fault happens in such a scenario. I think we have four options: abort only MPI_COMM_SELF (this seems too localized to me), abort all processors in the communicator (I don't think we can easily do this, since we don't have the communicator anymore, unless we change the behavior of free), abort all processes in MPI_COMM_WORLD (which seems too draconian), or to abort the immediate parent in the communicator hierarchy (which I am suggesting). An orthogonal issue is what to do, if we change the behavior of the error handler - do we follow the error handler of the actual, but freed communicator, or do we take the different handler of the parent. I don't have a good answer for this. Martin ` ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA On 6/1/15, 11:36 PM, "Fab Tillier" <ftillier(a)microsoft.com> wrote: >I think propagating upwards is weird, and only there to work around >poorly defined semantics at the cost of backward compatibility. > >Take an application that makes a dup of MPI_COMM_WORLD, then changes the >errhandler to MPI_ERRORS_RETURN. If this app then frees the dup >communicator handle after issuing an MPI_Irecv, it would currently likely >expect MPI_ERRORS_RETURN to be invoked if that receive encounters an >error, rather than MPI_ERRORS_ARE_FATAL that exists on MPI_COMM_WORLD. > >If an applications has a custom error handler that has per-communicator >state, it would need to either: >- keep track of outstanding requests on the communicator before freeing >the handle >- change the error handler to one of the built-in ones before freeing >- call MPI_COMM_DISCONNECT to ensure all pending communications are >complete (with potential ramifications for anything < MPI_THREAD_MULTIPLE) > >I think clarifying that an application that requires per-communicator >context in their error handler and calls MPI_COMM_FREE before all pending >operations are complete is erroneous is probably a better scoped solution >- MPI has never provided any mechanism to notify the application of when >a communicator is actually freed, and clearly defines that it may not be >freed when MPI_COMM_FREE returns. > >-Fab > >-----Original Message----- >From: mpiwg-ft [mailto:[email protected]] On Behalf Of >Schulz Martin >Sent: Tuesday, June 2, 2015 5:06 PM >To: MPI WG Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading > >Hi Wesley, > >Thanks for the summary - I think this describes it fairly well. > >One additional comment: there was some discussion on having to deal with >situations where there is/was an associated communicator, but that >communicator has been freed before the fault happened. The question was >which communicator an Abort should be on in this case. Personally, I think >it should propagate upwards to the next communicator in the hierarchy, >worst case to COMM_WORLD, but other options exist as well. It would be >good, though, to clearly define this case. > >Martin > > >________________________________________________________________________ >Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > >On 6/1/15, 10:00 PM, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > >>Notes from the ticket reading are now posted on the wiki: >>https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 >> >>TL;DR - The reading did not ³pass², but we got lots of good feedback to >>come back with a new version. We should consider splitting this into two >>or three tickets. One to define new errhandlers that does the new >>definition (abort communicator) and one that¹s a more well defined old >>definition (abort MPI_COMM_WORLD). Another ticket will deprecate >>MPI_COMM_ERRORS_ARE_FATAL. Another ticket will consolidate the >>definitions of all of the error handling text to a single place. >> >>The rest of the details can be found in the wiki. >> >>I¹ll be working on some drafts over the next few days to try to get new >>versions of this ticket out for discussion. My tentative hope is to get >>this ready for a new plenary in September. There¹s going to be enough >>changes that this should probably get a plenary before another reading. >> >>Comments welcome. >> >>Thanks, >>Wesley >>_______________________________________________ >>mpiwg-ft mailing list >>mpiwg-ft(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Ticket 324 June 2015 Reading
by Bland, Wesley 02 Jun '15

02 Jun '15

I think the thing that makes the most sense from a user’s perspective is to say that the error handler that was last set on the communicator when it was freed by the user will be called. I haven’t thought through all of the ramifications of how this could cause problems (of which I am sure there are many), but this seems to follow the law of least astonishment. If you set an error handler, you would expect it to be called. On the other hand, I’m all for telling people not to free their communicators until all of the requests attached to them are done, especially if you want proper error handling. Those are the two options that make the most sense to me. On June 2, 2015 at 12:27:13 AM, Fab Tillier (ftillier(a)microsoft.com<mailto:[email protected]>) wrote: Hi Martin, I think it's important to distinguish between communicator handle validity from the underlying communicator object's validity. You can think of MPI_COMM_FREE as freeing the handle and its reference on the underlying object (thus marking the communicator for deletion). Internally, the MPI implementation can keep the object around until all (other) references are freed (e.g. all pending operations complete). In case of an error, the implementation still has a valid object with which to call MPI_Abort, even if the user doesn't have a valid handle to that object (i.e. the user could not have called MPI_Abort themselves). The handle to such a communicator is effectively MPI_COMM_NULL. An application that wants higher fidelity for the handle being reported as the source should not have freed that communicator while operations were pending on it. I don't see why we need to define special behavior so that a user can make an equivalent call to MPI_Abort. -Fab -----Original Message----- From: mpiwg-ft [mailto:[email protected]] On Behalf Of Schulz Martin Sent: Tuesday, June 2, 2015 7:10 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading Hi Fab, all, I see where you are coming from and this makes sense to me. Defining this behavior more accurately and also defining what happens when a fault happens on freed communicator is clearly needed. The current standard text pretty much keeps the current communicator around, even if it is freed, and that's probably the right way to do it. However, the current text still treats it as freed and so we can't easily call Abort on it. So, the question is what to do with the Abort if a fault happens in such a scenario. I think we have four options: abort only MPI_COMM_SELF (this seems too localized to me), abort all processors in the communicator (I don't think we can easily do this, since we don't have the communicator anymore, unless we change the behavior of free), abort all processes in MPI_COMM_WORLD (which seems too draconian), or to abort the immediate parent in the communicator hierarchy (which I am suggesting). An orthogonal issue is what to do, if we change the behavior of the error handler - do we follow the error handler of the actual, but freed communicator, or do we take the different handler of the parent. I don't have a good answer for this. Martin ` ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA On 6/1/15, 11:36 PM, "Fab Tillier" <ftillier(a)microsoft.com> wrote: >I think propagating upwards is weird, and only there to work around >poorly defined semantics at the cost of backward compatibility. > >Take an application that makes a dup of MPI_COMM_WORLD, then changes the >errhandler to MPI_ERRORS_RETURN. If this app then frees the dup >communicator handle after issuing an MPI_Irecv, it would currently likely >expect MPI_ERRORS_RETURN to be invoked if that receive encounters an >error, rather than MPI_ERRORS_ARE_FATAL that exists on MPI_COMM_WORLD. > >If an applications has a custom error handler that has per-communicator >state, it would need to either: >- keep track of outstanding requests on the communicator before freeing >the handle >- change the error handler to one of the built-in ones before freeing >- call MPI_COMM_DISCONNECT to ensure all pending communications are >complete (with potential ramifications for anything < MPI_THREAD_MULTIPLE) > >I think clarifying that an application that requires per-communicator >context in their error handler and calls MPI_COMM_FREE before all pending >operations are complete is erroneous is probably a better scoped solution >- MPI has never provided any mechanism to notify the application of when >a communicator is actually freed, and clearly defines that it may not be >freed when MPI_COMM_FREE returns. > >-Fab > >-----Original Message----- >From: mpiwg-ft [mailto:[email protected]] On Behalf Of >Schulz Martin >Sent: Tuesday, June 2, 2015 5:06 PM >To: MPI WG Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading > >Hi Wesley, > >Thanks for the summary - I think this describes it fairly well. > >One additional comment: there was some discussion on having to deal with >situations where there is/was an associated communicator, but that >communicator has been freed before the fault happened. The question was >which communicator an Abort should be on in this case. Personally, I think >it should propagate upwards to the next communicator in the hierarchy, >worst case to COMM_WORLD, but other options exist as well. It would be >good, though, to clearly define this case. > >Martin > > >________________________________________________________________________ >Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > >On 6/1/15, 10:00 PM, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > >>Notes from the ticket reading are now posted on the wiki: >>https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 >> >>TL;DR - The reading did not ³pass², but we got lots of good feedback to >>come back with a new version. We should consider splitting this into two >>or three tickets. One to define new errhandlers that does the new >>definition (abort communicator) and one that¹s a more well defined old >>definition (abort MPI_COMM_WORLD). Another ticket will deprecate >>MPI_COMM_ERRORS_ARE_FATAL. Another ticket will consolidate the >>definitions of all of the error handling text to a single place. >> >>The rest of the details can be found in the wiki. >> >>I¹ll be working on some drafts over the next few days to try to get new >>versions of this ticket out for discussion. My tentative hope is to get >>this ready for a new plenary in September. There¹s going to be enough >>changes that this should probably get a plenary before another reading. >> >>Comments welcome. >> >>Thanks, >>Wesley >>_______________________________________________ >>mpiwg-ft mailing list >>mpiwg-ft(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Ticket 324 June 2015 Reading
by Bland, Wesley 02 Jun '15

02 Jun '15

It would get deprecated in favor of a similar error handler which would do similar, but slightly more well-defined things. The replacement would still abort the job. I would assume that we would want to make the replacement default error handler the one that aborts MPI_COMM_WORLD. In the end, the behavior would be exactly the same. It’s just that the “high quality implementation” specification would be slightly more specific. On June 1, 2015 at 10:00:12 PM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: Notes from the ticket reading are now posted on the wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 TL;DR - The reading did not “pass”, but we got lots of good feedback to come back with a new version. We should consider splitting this into two or three tickets. One to define new errhandlers that does the new definition (abort communicator) and one that’s a more well defined old definition (abort MPI_COMM_WORLD). Another ticket will deprecate MPI_COMM_ERRORS_ARE_FATAL. Another ticket will consolidate the definitions of all of the error handling text to a single place. The rest of the details can be found in the wiki. I’ll be working on some drafts over the next few days to try to get new versions of this ticket out for discussion. My tentative hope is to get this ready for a new plenary in September. There’s going to be enough changes that this should probably get a plenary before another reading. Comments welcome. Thanks, Wesley

1 0

Re: [mpiwg-ft] Ticket 324 June 2015 Reading
by Schulz Martin 02 Jun '15

02 Jun '15

Hi Fab, all, I see where you are coming from and this makes sense to me. Defining this behavior more accurately and also defining what happens when a fault happens on freed communicator is clearly needed. The current standard text pretty much keeps the current communicator around, even if it is freed, and that’s probably the right way to do it. However, the current text still treats it as freed and so we can’t easily call Abort on it. So, the question is what to do with the Abort if a fault happens in such a scenario. I think we have four options: abort only MPI_COMM_SELF (this seems too localized to me), abort all processors in the communicator (I don’t think we can easily do this, since we don’t have the communicator anymore, unless we change the behavior of free), abort all processes in MPI_COMM_WORLD (which seems too draconian), or to abort the immediate parent in the communicator hierarchy (which I am suggesting). An orthogonal issue is what to do, if we change the behavior of the error handler - do we follow the error handler of the actual, but freed communicator, or do we take the different handler of the parent. I don’t have a good answer for this. Martin ` ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA On 6/1/15, 11:36 PM, "Fab Tillier" <ftillier(a)microsoft.com> wrote: >I think propagating upwards is weird, and only there to work around >poorly defined semantics at the cost of backward compatibility. > >Take an application that makes a dup of MPI_COMM_WORLD, then changes the >errhandler to MPI_ERRORS_RETURN. If this app then frees the dup >communicator handle after issuing an MPI_Irecv, it would currently likely >expect MPI_ERRORS_RETURN to be invoked if that receive encounters an >error, rather than MPI_ERRORS_ARE_FATAL that exists on MPI_COMM_WORLD. > >If an applications has a custom error handler that has per-communicator >state, it would need to either: >- keep track of outstanding requests on the communicator before freeing >the handle >- change the error handler to one of the built-in ones before freeing >- call MPI_COMM_DISCONNECT to ensure all pending communications are >complete (with potential ramifications for anything < MPI_THREAD_MULTIPLE) > >I think clarifying that an application that requires per-communicator >context in their error handler and calls MPI_COMM_FREE before all pending >operations are complete is erroneous is probably a better scoped solution >- MPI has never provided any mechanism to notify the application of when >a communicator is actually freed, and clearly defines that it may not be >freed when MPI_COMM_FREE returns. > >-Fab > >-----Original Message----- >From: mpiwg-ft [mailto:[email protected]] On Behalf Of >Schulz Martin >Sent: Tuesday, June 2, 2015 5:06 PM >To: MPI WG Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading > >Hi Wesley, > >Thanks for the summary - I think this describes it fairly well. > >One additional comment: there was some discussion on having to deal with >situations where there is/was an associated communicator, but that >communicator has been freed before the fault happened. The question was >which communicator an Abort should be on in this case. Personally, I think >it should propagate upwards to the next communicator in the hierarchy, >worst case to COMM_WORLD, but other options exist as well. It would be >good, though, to clearly define this case. > >Martin > > >________________________________________________________________________ >Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > >On 6/1/15, 10:00 PM, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > >>Notes from the ticket reading are now posted on the wiki: >>https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 >> >>TL;DR - The reading did not ³pass², but we got lots of good feedback to >>come back with a new version. We should consider splitting this into two >>or three tickets. One to define new errhandlers that does the new >>definition (abort communicator) and one that¹s a more well defined old >>definition (abort MPI_COMM_WORLD). Another ticket will deprecate >>MPI_COMM_ERRORS_ARE_FATAL. Another ticket will consolidate the >>definitions of all of the error handling text to a single place. >> >>The rest of the details can be found in the wiki. >> >>I¹ll be working on some drafts over the next few days to try to get new >>versions of this ticket out for discussion. My tentative hope is to get >>this ready for a new plenary in September. There¹s going to be enough >>changes that this should probably get a plenary before another reading. >> >>Comments welcome. >> >>Thanks, >>Wesley >>_______________________________________________ >>mpiwg-ft mailing list >>mpiwg-ft(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Ticket 324 June 2015 Reading
by Fab Tillier 02 Jun '15

02 Jun '15

Hi Martin, I think it's important to distinguish between communicator handle validity from the underlying communicator object's validity. You can think of MPI_COMM_FREE as freeing the handle and its reference on the underlying object (thus marking the communicator for deletion). Internally, the MPI implementation can keep the object around until all (other) references are freed (e.g. all pending operations complete). In case of an error, the implementation still has a valid object with which to call MPI_Abort, even if the user doesn't have a valid handle to that object (i.e. the user could not have called MPI_Abort themselves). The handle to such a communicator is effectively MPI_COMM_NULL. An application that wants higher fidelity for the handle being reported as the source should not have freed that communicator while operations were pending on it. I don't see why we need to define special behavior so that a user can make an equivalent call to MPI_Abort. -Fab -----Original Message----- From: mpiwg-ft [mailto:[email protected]] On Behalf Of Schulz Martin Sent: Tuesday, June 2, 2015 7:10 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading Hi Fab, all, I see where you are coming from and this makes sense to me. Defining this behavior more accurately and also defining what happens when a fault happens on freed communicator is clearly needed. The current standard text pretty much keeps the current communicator around, even if it is freed, and that's probably the right way to do it. However, the current text still treats it as freed and so we can't easily call Abort on it. So, the question is what to do with the Abort if a fault happens in such a scenario. I think we have four options: abort only MPI_COMM_SELF (this seems too localized to me), abort all processors in the communicator (I don't think we can easily do this, since we don't have the communicator anymore, unless we change the behavior of free), abort all processes in MPI_COMM_WORLD (which seems too draconian), or to abort the immediate parent in the communicator hierarchy (which I am suggesting). An orthogonal issue is what to do, if we change the behavior of the error handler - do we follow the error handler of the actual, but freed communicator, or do we take the different handler of the parent. I don't have a good answer for this. Martin ` ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA On 6/1/15, 11:36 PM, "Fab Tillier" <ftillier(a)microsoft.com> wrote: >I think propagating upwards is weird, and only there to work around >poorly defined semantics at the cost of backward compatibility. > >Take an application that makes a dup of MPI_COMM_WORLD, then changes the >errhandler to MPI_ERRORS_RETURN. If this app then frees the dup >communicator handle after issuing an MPI_Irecv, it would currently likely >expect MPI_ERRORS_RETURN to be invoked if that receive encounters an >error, rather than MPI_ERRORS_ARE_FATAL that exists on MPI_COMM_WORLD. > >If an applications has a custom error handler that has per-communicator >state, it would need to either: >- keep track of outstanding requests on the communicator before freeing >the handle >- change the error handler to one of the built-in ones before freeing >- call MPI_COMM_DISCONNECT to ensure all pending communications are >complete (with potential ramifications for anything < MPI_THREAD_MULTIPLE) > >I think clarifying that an application that requires per-communicator >context in their error handler and calls MPI_COMM_FREE before all pending >operations are complete is erroneous is probably a better scoped solution >- MPI has never provided any mechanism to notify the application of when >a communicator is actually freed, and clearly defines that it may not be >freed when MPI_COMM_FREE returns. > >-Fab > >-----Original Message----- >From: mpiwg-ft [mailto:[email protected]] On Behalf Of >Schulz Martin >Sent: Tuesday, June 2, 2015 5:06 PM >To: MPI WG Fault Tolerance and Dynamic Process Control working Group >Subject: Re: [mpiwg-ft] Ticket 324 June 2015 Reading > >Hi Wesley, > >Thanks for the summary - I think this describes it fairly well. > >One additional comment: there was some discussion on having to deal with >situations where there is/was an associated communicator, but that >communicator has been freed before the fault happened. The question was >which communicator an Abort should be on in this case. Personally, I think >it should propagate upwards to the next communicator in the hierarchy, >worst case to COMM_WORLD, but other options exist as well. It would be >good, though, to clearly define this case. > >Martin > > >________________________________________________________________________ >Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > >On 6/1/15, 10:00 PM, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > >>Notes from the ticket reading are now posted on the wiki: >>https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2015-06-01 >> >>TL;DR - The reading did not ³pass², but we got lots of good feedback to >>come back with a new version. We should consider splitting this into two >>or three tickets. One to define new errhandlers that does the new >>definition (abort communicator) and one that¹s a more well defined old >>definition (abort MPI_COMM_WORLD). Another ticket will deprecate >>MPI_COMM_ERRORS_ARE_FATAL. Another ticket will consolidate the >>definitions of all of the error handling text to a single place. >> >>The rest of the details can be found in the wiki. >> >>I¹ll be working on some drafts over the next few days to try to get new >>versions of this ticket out for discussion. My tentative hope is to get >>this ready for a new plenary in September. There¹s going to be enough >>changes that this should probably get a plenary before another reading. >> >>Comments welcome. >> >>Thanks, >>Wesley >>_______________________________________________ >>mpiwg-ft mailing list >>mpiwg-ft(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >_______________________________________________ >mpiwg-ft mailing list >mpiwg-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0