August 2020 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] MPI_Request_free restrictions
by Jim Dinan 13 Aug '20

13 Aug '20

The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. ~Jim. On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum < mpi-forum(a)lists.mpi-forum.org> wrote: > Hi all, > > To increase my own understanding of RMA, what is the difference (if any) > between a request-based RMA operation where the request is freed without > being completed and before the epoch is closed and a “normal” RMA operation? > > MPI_LOCK() ! or any other "open epoch at origin" procedure call > doUserWorkBefore() > MPI_RPUT(&req) > MPI_REQUEST_FREE(&req) > doUserWorkAfter() > MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call > > vs: > > MPI_LOCK() ! or any other "open epoch at origin" procedure call > doUserWorkBefore() > MPI_PUT() > doUserWorkAfter() > MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call > > Is this a source-to-source translation that is always safe in either > direction? > > In RMA, in contrast to the rest of MPI, there are two opportunities for > MPI to “block” and do non-local work to complete an RMA operation: 1) > during MPI_WAIT for the request (if any - the user may not be given a > request or the user may choose to free the request without calling MPI_WAIT > or the user might call nonblocking MPI_TEST) and 2) during the close epoch > procedure, which is always permitted to be sufficiently non-local to > guarantee that the RMA operation is complete and its freeing stage has been > done. It seems that a request-based RMA operation becomes identical to a > “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. > This is like “freeing" the request from a nonblocking point-to-point > operation but without the guarantee of a later synchronisation procedure > that can actually complete the operation and actually do the freeing stage > of the operation. > > In collectives, there is no “ensure all operations so far are now done” > procedure call because there is no concept of epoch for collectives. > In point-to-point, there is no “ensure all operations so far are now done” > procedure call because there is no concept of epoch for point-to-point. > In file operations, there is no “ensure all operations so far are now > done” procedure call because there is no concept of epoch for file > operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely > on it being called.) > In these cases, the only non-local procedure that is guaranteed to happen > is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” > operation might be delayed until that procedure is called. > > The issue with copying parameters is also moot because all of them are > passed-by-value (implicitly copied) or are data-buffers and covered by > “conflicting accesses” RMA rules. > > Thus, to me it seems to me that RMA is a very special case - it could > support different semantics, but that does not provide a good basis for > claiming that the rest of the MPI Standard can support those different > semantics - unless we introduce an epoch concept into the rest of the MPI > Standard. This is not unreasonable: the notifications in GASPI, for > example, guarantee completion of not just the operation they are attached > to but *all* operations issued in the “queue” they represent since the last > notification. Their queue concept serves the purpose of an epoch. I’m sure > there are other examples in other APIs. It seems to me likely that the > proposal for MPI_PSYNC for partitioned communication operations is moving > in the direction of an epoch, although limited to remote completion of all > the partitions in a single operation, which accidentally guarantees that > the operation can be freed locally using a local procedure. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Architect (HPC Research) > d.holmes(a)epcc.ed.ac.uk > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, > EH8 9BT > — > The University of Edinburgh is a charitable body, registered in Scotland, > with registration number SC005336. > — > > On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum < > mpi-forum(a)lists.mpi-forum.org> wrote: > > FYI, one argument (also used to force us to add restrictions on MPI > persistent collective initialization to be blocking)... The > MPI_Request_free on an NBC poses a problem for the cases where there are > array types > posed (e.g., Alltoallv/w)... It will not be knowable to the application if > the vectors are in use by MPI still after > the free on an active request. We do *not* mandate that the MPI > implementation copy such arrays currently, so they are effectively "held as > unfreeable" by the MPI implementation till MPI_Finalize. The user cannot > deallocate them in a correct program till after MPI_Finalize. > > Another effect for NBC of releasing an active request, IMHO, is that you > don't know when send buffers are free to be deallocated or receive buffers > are free to be deallocated... since you don't know when the transfer is > complete OR the buffers are no longer used by MPI (till after MPI_Finalize). > > Tony > > > > > Anthony Skjellum, PhD > Professor of Computer Science and Chair of Excellence > Director, SimCenter > University of Tennessee at Chattanooga (UTC) > tony-skjellum(a)utc.edu [or skjellum(a)gmail.com] > cell: 205-807-4968 > > ------------------------------ > *From:* mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of > Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org> > *Sent:* Saturday, August 8, 2020 12:07 PM > *To:* Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> > *Cc:* Jeff Hammond <jeff.science(a)gmail.com> > *Subject:* Re: [Mpi-forum] MPI_Request_free restrictions > > We should fix the RMA chapter with an erratum. I care less about NBC but > share your ignorance of why it was done that way. > > Sent from my iPhone > > On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum < > mpi-forum(a)lists.mpi-forum.org> wrote: > > Folks, > > Does someone remember why we disallowed users from calling > MPI_Request_free on nonblocking collective requests? I remember the > reasoning for not allowing cancel (i.e., the operation might have completed > on some processes, but not all), but not for Request_free. AFAICT, > allowing the users to free the request doesn’t make any difference to the > MPI library. The MPI library would simply maintain its own refcount to the > request and continue forward till the operation completes. One of our > users would like to free NBC requests so they don’t have to wait for the > operation to complete in some situations. > > Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I > copy-pasted that text into RMA as well without thinking too hard about it. > My bad! Either the RMA committee missed it too, or they thought of a > reason that I can’t think of now. > > Can someone clarify or remind me what the reason was? > > Regards, > > — Pavan > > MPI-3.1 standard, page 197, lines 26-27: > > “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request > associated with a nonblocking collective operation.” > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by HOLMES Daniel 13 Aug '20

13 Aug '20

Hi all, To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_RPUT(&req) MPI_REQUEST_FREE(&req) doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call vs: MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_PUT() doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call Is this a source-to-source translation that is always safe in either direction? In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). Tony Anthony Skjellum, PhD Professor of Computer Science and Chair of Excellence Director, SimCenter University of Tennessee at Chattanooga (UTC) tony-skjellum(a)utc.edu<mailto:[email protected]> [or skjellum(a)gmail.com<mailto:[email protected]>] cell: 205-807-4968 ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: Saturday, August 8, 2020 12:07 PM To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. Sent from my iPhone On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Folks, Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. Can someone clarify or remind me what the reason was? Regards, — Pavan MPI-3.1 standard, page 197, lines 26-27: “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Skjellum, Anthony 12 Aug '20

12 Aug '20

FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). Tony Anthony Skjellum, PhD Professor of Computer Science and Chair of Excellence Director, SimCenter University of Tennessee at Chattanooga (UTC) tony-skjellum(a)utc.edu [or skjellum(a)gmail.com] cell: 205-807-4968 ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org> Sent: Saturday, August 8, 2020 12:07 PM To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> Cc: Jeff Hammond <jeff.science(a)gmail.com> Subject: Re: [Mpi-forum] MPI_Request_free restrictions We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. Sent from my iPhone On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: Folks, Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. Can someone clarify or remind me what the reason was? Regards, — Pavan MPI-3.1 standard, page 197, lines 26-27: “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Jeff Hammond 12 Aug '20

12 Aug '20

Does anybody implement RMA and NBC requests differently from P2P ones such that this matters? How would an implementation support a different type of request for NBC+RMA than P2P? If an MPI_Request has to be a reference for P2P, how is it going to also support the not-a-reference case? I can imagine a union of P2P_Reference, RMA_Object_itself, and NBC_Object_itself, but seems like it would blow up the memory requirements for a vector of MPI_Requests if RMA_Object_itself or NBC_Object_itself are larger than 8 bytes. I wonder if it is even a good idea to not maintain a reference to the request. In implementations that do not implement passive progress on everything, how does the progress engine know to poke an NBC request if there isn't an implementation of it in the queue? Jeff On Wed, Aug 12, 2020 at 1:51 PM Jim Dinan <james.dinan(a)gmail.com> wrote: > IIRC, the argument for RMA was that we had not wanted to require the MPI > library to maintain a reference to the request. There could be many RMA > operations pending and the MPI library maintaining a list of pending > operations could be a significant source of overhead. I seem to recall this > having been the general sentiment at the time and it may have also applied > to NBC. > > ~Jim. > > On Sat, Aug 8, 2020 at 7:15 PM Jeff Hammond via mpi-forum < > mpi-forum(a)lists.mpi-forum.org> wrote: > >> We should fix the RMA chapter with an erratum. I care less about NBC but >> share your ignorance of why it was done that way. >> >> Sent from my iPhone >> >> On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum < >> mpi-forum(a)lists.mpi-forum.org> wrote: >> >> Folks, >> >> Does someone remember why we disallowed users from calling >> MPI_Request_free on nonblocking collective requests? I remember the >> reasoning for not allowing cancel (i.e., the operation might have completed >> on some processes, but not all), but not for Request_free. AFAICT, >> allowing the users to free the request doesn’t make any difference to the >> MPI library. The MPI library would simply maintain its own refcount to the >> request and continue forward till the operation completes. One of our >> users would like to free NBC requests so they don’t have to wait for the >> operation to complete in some situations. >> >> Unfortunately, when I added the Rput/Rget operations in the RMA chapter, >> I copy-pasted that text into RMA as well without thinking too hard about >> it. My bad! Either the RMA committee missed it too, or they thought of a >> reason that I can’t think of now. >> >> Can someone clarify or remind me what the reason was? >> >> Regards, >> >> — Pavan >> >> MPI-3.1 standard, page 197, lines 26-27: >> >> “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request >> associated with a nonblocking collective operation.” >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Jim Dinan 12 Aug '20

12 Aug '20

IIRC, the argument for RMA was that we had not wanted to require the MPI library to maintain a reference to the request. There could be many RMA operations pending and the MPI library maintaining a list of pending operations could be a significant source of overhead. I seem to recall this having been the general sentiment at the time and it may have also applied to NBC. ~Jim. On Sat, Aug 8, 2020 at 7:15 PM Jeff Hammond via mpi-forum < mpi-forum(a)lists.mpi-forum.org> wrote: > We should fix the RMA chapter with an erratum. I care less about NBC but > share your ignorance of why it was done that way. > > Sent from my iPhone > > On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum < > mpi-forum(a)lists.mpi-forum.org> wrote: > > Folks, > > Does someone remember why we disallowed users from calling > MPI_Request_free on nonblocking collective requests? I remember the > reasoning for not allowing cancel (i.e., the operation might have completed > on some processes, but not all), but not for Request_free. AFAICT, > allowing the users to free the request doesn’t make any difference to the > MPI library. The MPI library would simply maintain its own refcount to the > request and continue forward till the operation completes. One of our > users would like to free NBC requests so they don’t have to wait for the > operation to complete in some situations. > > Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I > copy-pasted that text into RMA as well without thinking too hard about it. > My bad! Either the RMA committee missed it too, or they thought of a > reason that I can’t think of now. > > Can someone clarify or remind me what the reason was? > > Regards, > > — Pavan > > MPI-3.1 standard, page 197, lines 26-27: > > “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request > associated with a nonblocking collective operation.” > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Jeff Hammond 08 Aug '20

08 Aug '20

Correct, but that’s not what I was suggesting. I was saying that the root alone could track completion of any number of collectives individually and then broadcast notice of those completions to all ranks. This is but one example use case where freeing requests associated with NBCs makes sense. Jeff Sent from my iPhone > On Aug 8, 2020, at 2:14 PM, Richard Graham <richgraham(a)nvidia.com> wrote: > > > The non-blocking reductions are required to start in the same order, completion order is not specified. So I don’t think you can infer from the completion of the last non-blocking operation (ibarrier in this case), that all others have completed. > > Also, completion is local, and even in a synchronized operation like a reduction, not all ranks will complete at exactly the same time. All have to enter the operation before any can complete, but the fact that one completed does not guarantee the other have. > > Rich > > From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> On Behalf Of Jeff Hammond via mpi-forum > Sent: Saturday, August 8, 2020 3:56 PM > To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> > Cc: Jeff Hammond <jeff.science(a)gmail.com> > Subject: Re: [Mpi-forum] MPI_Request_free restrictions > > External email: Use caution opening links or attachments > > The argument that there is no portable way to detect completion is false. It is completely portable to detect completion using only a subset of processes, and there may even be use cases for it. > > For example, an application can use Ibarrier to determine whether all processes have reached a phase of the program but only detect this at one rank. > > MPI_Request req; > { > foo(); > MPI_Ibarrier(comm, &req); > if (rank) MPI_Request_free(&req); > .. > } > if (!rank) { > MPI_Wait(&req); > printf("foo finished everywhere\n"); > } > > Similarly, one might have a bunch of Ireduce operations where the root is the only rank that tests completion, after which it notifies all processes that the bunch has completed. > > Are there any implementations that can't trivially support this? I recall that MPICH implements requests as nothing more than pointers to internal state, and I assume that Open-MPI does something similar. > > Jeff > > On Sat, Aug 8, 2020 at 9:45 AM Bangalore, Purushotham via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: > I see discussion of this issue here: > > https://github.com/mpi-forum/mpi-forum-historic/issues/83 > > Puri > From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org> > Sent: Saturday, August 8, 2020 8:51 AM > To: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> > Cc: Balaji, Pavan <balaji(a)anl.gov> > Subject: [Mpi-forum] MPI_Request_free restrictions > > Folks, > > Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. > > Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. > > Can someone clarify or remind me what the reason was? > > Regards, > > — Pavan > > MPI-3.1 standard, page 197, lines 26-27: > > “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Jeff Hammond 08 Aug '20

08 Aug '20

The argument that there is no portable way to detect completion is false. It is completely portable to detect completion using only a subset of processes, and there may even be use cases for it. For example, an application can use Ibarrier to determine whether all processes have reached a phase of the program but only detect this at one rank. MPI_Request req; { foo(); MPI_Ibarrier(comm, &req); if (rank) MPI_Request_free(&req); .. } if (!rank) { MPI_Wait(&req); printf("foo finished everywhere\n"); } Similarly, one might have a bunch of Ireduce operations where the root is the only rank that tests completion, after which it notifies all processes that the bunch has completed. Are there any implementations that can't trivially support this? I recall that MPICH implements requests as nothing more than pointers to internal state, and I assume that Open-MPI does something similar. Jeff On Sat, Aug 8, 2020 at 9:45 AM Bangalore, Purushotham via mpi-forum < mpi-forum(a)lists.mpi-forum.org> wrote: > I see discussion of this issue here: > > https://github.com/mpi-forum/mpi-forum-historic/issues/83 > > Puri > ------------------------------ > *From:* mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of > Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org> > *Sent:* Saturday, August 8, 2020 8:51 AM > *To:* mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> > *Cc:* Balaji, Pavan <balaji(a)anl.gov> > *Subject:* [Mpi-forum] MPI_Request_free restrictions > > Folks, > > Does someone remember why we disallowed users from calling > MPI_Request_free on nonblocking collective requests? I remember the > reasoning for not allowing cancel (i.e., the operation might have completed > on some processes, but not all), but not for Request_free. AFAICT, > allowing the users to free the request doesn’t make any difference to the > MPI library. The MPI library would simply maintain its own refcount to the > request and continue forward till the operation completes. One of our > users would like to free NBC requests so they don’t have to wait for the > operation to complete in some situations. > > Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I > copy-pasted that text into RMA as well without thinking too hard about it. > My bad! Either the RMA committee missed it too, or they thought of a > reason that I can’t think of now. > > Can someone clarify or remind me what the reason was? > > Regards, > > — Pavan > > MPI-3.1 standard, page 197, lines 26-27: > > “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request > associated with a nonblocking collective operation.” > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by George Bosilca 08 Aug '20

08 Aug '20

Similar discussion can be found in the "advice to users" in 3.7.3 (page 63 line 18). George. On Sat, Aug 8, 2020 at 12:45 PM Bangalore, Purushotham via mpi-forum < mpi-forum(a)lists.mpi-forum.org> wrote: > I see discussion of this issue here: > > https://github.com/mpi-forum/mpi-forum-historic/issues/83 > > Puri > ------------------------------ > *From:* mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of > Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org> > *Sent:* Saturday, August 8, 2020 8:51 AM > *To:* mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> > *Cc:* Balaji, Pavan <balaji(a)anl.gov> > *Subject:* [Mpi-forum] MPI_Request_free restrictions > > Folks, > > Does someone remember why we disallowed users from calling > MPI_Request_free on nonblocking collective requests? I remember the > reasoning for not allowing cancel (i.e., the operation might have completed > on some processes, but not all), but not for Request_free. AFAICT, > allowing the users to free the request doesn’t make any difference to the > MPI library. The MPI library would simply maintain its own refcount to the > request and continue forward till the operation completes. One of our > users would like to free NBC requests so they don’t have to wait for the > operation to complete in some situations. > > Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I > copy-pasted that text into RMA as well without thinking too hard about it. > My bad! Either the RMA committee missed it too, or they thought of a > reason that I can’t think of now. > > Can someone clarify or remind me what the reason was? > > Regards, > > — Pavan > > MPI-3.1 standard, page 197, lines 26-27: > > “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request > associated with a nonblocking collective operation.” > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Bangalore, Purushotham 08 Aug '20

08 Aug '20

I see discussion of this issue here: https://github.com/mpi-forum/mpi-forum-historic/issues/83 Puri ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org> Sent: Saturday, August 8, 2020 8:51 AM To: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Cc: Balaji, Pavan <balaji(a)anl.gov> Subject: [Mpi-forum] MPI_Request_free restrictions Folks, Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. Can someone clarify or remind me what the reason was? Regards, — Pavan MPI-3.1 standard, page 197, lines 26-27: “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.”

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Jeff Hammond 08 Aug '20

08 Aug '20

We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. Sent from my iPhone > On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: > > Folks, > > Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. > > Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. > > Can someone clarify or remind me what the reason was? > > Regards, > > — Pavan > > MPI-3.1 standard, page 197, lines 26-27: > > “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0