August 2020 - mpi-forum - mailman3.cels.anl.gov

Reminder: Virtual MPI Forum Meeting tomorrow (Wednesday) 10am US Central Time
by Martin Schulz 25 Aug '20

25 Aug '20

Hi all, Just as a reminder - due to the short gap between the two voting meetings in August and September, we decided to already use this week for the next virtual discussions again. Therefore, the next meeting is tomorrow (Wednesday) at the usual time 10am US Central. The link is, as always, linked from the MPI Forum website. We have two items on the agenda: - Change requests for the BigCount ticket (#137) - "The table" covering T&C properties for each MPI function (used to be #96) Talk to you tomorrow, Thanks! Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by HOLMES Daniel 24 Aug '20

24 Aug '20

Hi Quincey, Puri unearthed the Github issues that are proposing this/these change(s) - thanks, Puri! Reading through the comments on both, it seems to me that #78 is the over-arching container for our thoughts and discussions. https://github.com/mpi-forum/mpi-issues/issues/78 (You could also create a blog post or a paper and reference that from the issue. A peer reviewed paper supporting the interface change guarantees greater traction, but it is substantially more work.) Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 22 Aug 2020, at 20:48, Quincey Koziol <koziol(a)lbl.gov<mailto:[email protected]>> wrote: Hi Dan, On Aug 21, 2020, at 10:48 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, The idea of adding MPI_COMM_IDISCONNECT (also MPI_FILE_ISYNC, as I should have mentioned earlier) is already on the to-do list. Calling MPI_COMM_DISCONNECT replaces the call to MPI_COMM_FREE (both set the comm argument to MPI_COMM_NULL and end the user’s direct interaction/manipulation with/of that communicator). Having called either of these two procedures, the user must not call the other one - and cannot because they no longer have a valid handle for the communicator. Got it. That means that MPI_COMM_IDISCONNECT would be somewhat similar to MPI_COMM_FREE (except for the requirement on the user to have already called the completion procedures for pending operations, and it is unlikely to be collective - it will be local, I hope) but, when followed by MPI_WAIT for the request it created as output, it would provide the guarantee given by MPI_COMM_DISCONNECT but not given by MPI_COMM_FREE, i.e. that all pending communication using that communicator has now actually finished internally. Using MPI_TEST permits an entirely nonblocking usage pattern that polls until this guarantee can be made by MPI. Both of these are anticipated to be useful for applications. It would be good to capture your use-case more precisely so that it can be used to justify these changes when they eventually come up for a vote in the MPI Forum. Sure - where should I document the HDF5 use cases for these routines? Quincey Personally, I believe that MPI_COMM_IDISCONNECT (once it exists) should entirely supplant MPI_COMM_FREE, but that is still somewhat a contentious point-of-view. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 21 Aug 2020, at 16:19, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, Ah, very useful to know, thanks! Is there a nonblocking version of MPI_COMM_DISCONNECT? (I’ve searched the web for MPI_COMM_IDISCONNECT and it comes up empty, but that’s not canonical :-) If not, can a capability like this be added to any “wish lists”? Ideally, calling something like MPI_COMM_IDISCONNECT and then having the request for that operation complete would mean that MPI_COMM_FREE would be guaranteed to be both nonblocking and complete locally. Thoughts? Quincey On Aug 21, 2020, at 10:01 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, Calling MPI_COMM_FREE when some requests representing nonblocking or persistent operations are still active is not prohibited by MPI and seems to work successfully in all the MPI libraries I’ve tested. The normative description for MPI_COMM_FREE in the MPI Standard specifically calls out that it will only mark the communicator for freeing later and may return to the user before pending/ongoing communication is complete. It does not require that the completion procedure has been called for active operations. We discussed in the Forum (as recently as the meeting this week) that this is a key difference between MPI_COMM_FREE and MPI_COMM_DISCONNECT - the latter states that the user is required to call the completion procedure(s) for all operations using a communicator before disconnecting it using MPI_COMM_DISCONNECT, which will wait for all pending communication to complete internally. OTOH, I’m not sure that doing this buys you as much as you think it might. MPI_COMM_FREE is a collective procedure, so it is permitted to wait until MPI_COMM_FREE has been called at all other MPI processes in the communicator, i.e. it can have blocking-barrier-like semantics. All collective operations must be initialised in the same order at all processes in the communicator. So a valid implementation could do all the pending work inside MPI_COMM_FREE but the Standard also permits an implementation that does nothing other than change a “ready-for-freeing” flag on the local communicator object. Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Yes. Will MPI_COMM_FREE block for completion of the NBC op? No. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 21 Aug 2020, at 15:26, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I agree with you about MPI barriers, but that’s why I said it was a simplified pseudocode. :-) We do have more mechanisms in place for handling the "fence-ness” of the operation, but barriers are a component and I’d like to move to a nonblocking version when possible. Having more, hopefully all, of the file operations get equivalent nonblocking versions would be _very_ nice, and I could simplify our internal code more if a nonblocking MPI_FILE_SYNC was available. A nonblocking version of MPI_FILE_SET_SIZE would also be high on my list. Yes, I grok the behavior of MPI_FILE_CLOSE, but don’t want to add a barrier on top of it. :-) One new question: Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Will MPI_COMM_FREE block for completion of the NBC op? Thanks! Quincey On Aug 15, 2020, at 6:07 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files). Perhaps you are looking for MPI_FILE_SYNC? "If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1 "MPI_FILE_SYNC is a collective operation.” §13.6.1 Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally. There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next. As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do: =============================== <open HDF5 file> // sets up a communicator for internal HDF5 communication about this file do { MPI_Ibarrier(<file’s communicator>, &request); <application stuff> // HDF5 operation: if(<operation is read or write>) { MPI_Wait(&request); <perform read / write> } else { // operation is a file close MPI_Request_free(&request); MPI_File_close(…); MPI_Comm_free(<file’s communicator>); } } while (<file is open>); =============================== What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed of when the nonblocking barrier completes locally. So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request. Thoughts? Quincey On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Jim, To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience. I am certainly not arguing that it be permitted for more MPI operations. I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request. If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE without copying it. As you say, MPI takes over the responsibility for processing the completion event. Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer to take on this responsibility in the first place?” Thanks, that helps (me at least). Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 14:43, Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> wrote: The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. ~Jim. On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_RPUT(&req) MPI_REQUEST_FREE(&req) doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call vs: MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_PUT() doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call Is this a source-to-source translation that is always safe in either direction? In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). Tony Anthony Skjellum, PhD Professor of Computer Science and Chair of Excellence Director, SimCenter University of Tennessee at Chattanooga (UTC) tony-skjellum(a)utc.edu<mailto:[email protected]> [or skjellum(a)gmail.com<mailto:[email protected]>] cell: 205-807-4968 ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: Saturday, August 8, 2020 12:07 PM To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. Sent from my iPhone On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Folks, Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. Can someone clarify or remind me what the reason was? Regards, — Pavan MPI-3.1 standard, page 197, lines 26-27: “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Quincey Koziol 22 Aug '20

22 Aug '20

Hi Dan, > On Aug 21, 2020, at 10:48 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> wrote: > > Hi Quincey, > > The idea of adding MPI_COMM_IDISCONNECT (also MPI_FILE_ISYNC, as I should have mentioned earlier) is already on the to-do list. > > Calling MPI_COMM_DISCONNECT replaces the call to MPI_COMM_FREE (both set the comm argument to MPI_COMM_NULL and end the user’s direct interaction/manipulation with/of that communicator). Having called either of these two procedures, the user must not call the other one - and cannot because they no longer have a valid handle for the communicator. Got it. > That means that MPI_COMM_IDISCONNECT would be somewhat similar to MPI_COMM_FREE (except for the requirement on the user to have already called the completion procedures for pending operations, and it is unlikely to be collective - it will be local, I hope) but, when followed by MPI_WAIT for the request it created as output, it would provide the guarantee given by MPI_COMM_DISCONNECT but not given by MPI_COMM_FREE, i.e. that all pending communication using that communicator has now actually finished internally. Using MPI_TEST permits an entirely nonblocking usage pattern that polls until this guarantee can be made by MPI. Both of these are anticipated to be useful for applications. It would be good to capture your use-case more precisely so that it can be used to justify these changes when they eventually come up for a vote in the MPI Forum. Sure - where should I document the HDF5 use cases for these routines? Quincey > > Personally, I believe that MPI_COMM_IDISCONNECT (once it exists) should entirely supplant MPI_COMM_FREE, but that is still somewhat a contentious point-of-view. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Architect (HPC Research) > d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT > — > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > — > >> On 21 Aug 2020, at 16:19, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >> >> Hi Dan, >> Ah, very useful to know, thanks! Is there a nonblocking version of MPI_COMM_DISCONNECT? (I’ve searched the web for MPI_COMM_IDISCONNECT and it comes up empty, but that’s not canonical :-) >> >> If not, can a capability like this be added to any “wish lists”? Ideally, calling something like MPI_COMM_IDISCONNECT and then having the request for that operation complete would mean that MPI_COMM_FREE would be guaranteed to be both nonblocking and complete locally. Thoughts? >> >> Quincey >> >> >>> On Aug 21, 2020, at 10:01 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]>> wrote: >>> >>> Hi Quincey, >>> >>> Calling MPI_COMM_FREE when some requests representing nonblocking or persistent operations are still active is not prohibited by MPI and seems to work successfully in all the MPI libraries I’ve tested. >>> >>> The normative description for MPI_COMM_FREE in the MPI Standard specifically calls out that it will only mark the communicator for freeing later and may return to the user before pending/ongoing communication is complete. It does not require that the completion procedure has been called for active operations. >>> >>> We discussed in the Forum (as recently as the meeting this week) that this is a key difference between MPI_COMM_FREE and MPI_COMM_DISCONNECT - the latter states that the user is required to call the completion procedure(s) for all operations using a communicator before disconnecting it using MPI_COMM_DISCONNECT, which will wait for all pending communication to complete internally. >>> >>> OTOH, I’m not sure that doing this buys you as much as you think it might. >>> >>> MPI_COMM_FREE is a collective procedure, so it is permitted to wait until MPI_COMM_FREE has been called at all other MPI processes in the communicator, i.e. it can have blocking-barrier-like semantics. All collective operations must be initialised in the same order at all processes in the communicator. So a valid implementation could do all the pending work inside MPI_COMM_FREE but the Standard also permits an implementation that does nothing other than change a “ready-for-freeing” flag on the local communicator object. >>> >>>> Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? >>> >>> Yes. >>> >>>> Will MPI_COMM_FREE block for completion of the NBC op? >>> >>> No. >>> >>> Cheers, >>> Dan. >>> — >>> Dr Daniel Holmes PhD >>> Architect (HPC Research) >>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>> Phone: +44 (0) 131 651 3465 >>> Mobile: +44 (0) 7940 524 088 >>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>> — >>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>> — >>> >>>> On 21 Aug 2020, at 15:26, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>> >>>> Hi Dan, >>>> I agree with you about MPI barriers, but that’s why I said it was a simplified pseudocode. :-) We do have more mechanisms in place for handling the "fence-ness” of the operation, but barriers are a component and I’d like to move to a nonblocking version when possible. >>>> >>>> Having more, hopefully all, of the file operations get equivalent nonblocking versions would be _very_ nice, and I could simplify our internal code more if a nonblocking MPI_FILE_SYNC was available. A nonblocking version of MPI_FILE_SET_SIZE would also be high on my list. >>>> >>>> Yes, I grok the behavior of MPI_FILE_CLOSE, but don’t want to add a barrier on top of it. :-) >>>> >>>> >>>> One new question: Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Will MPI_COMM_FREE block for completion of the NBC op? >>>> >>>> >>>> Thanks! >>>> Quincey >>>> >>>> >>>> >>>>> On Aug 15, 2020, at 6:07 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]>> wrote: >>>>> >>>>> Hi Quincey, >>>>> >>>>> The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files). >>>>> >>>>> Perhaps you are looking for MPI_FILE_SYNC? >>>>> >>>>> "If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1 >>>>> >>>>> "MPI_FILE_SYNC is a collective operation.” §13.6.1 >>>>> >>>>> Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally. >>>>> >>>>> There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next. >>>>> >>>>> As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called. >>>>> >>>>> Cheers, >>>>> Dan. >>>>> — >>>>> Dr Daniel Holmes PhD >>>>> Architect (HPC Research) >>>>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>>>> Phone: +44 (0) 131 651 3465 >>>>> Mobile: +44 (0) 7940 524 088 >>>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>>>> — >>>>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>>>> — >>>>> >>>>>> On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>> >>>>>> Hi Dan, >>>>>> I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do: >>>>>> >>>>>> =============================== >>>>>> >>>>>> <open HDF5 file> // sets up a communicator for internal HDF5 communication about this file >>>>>> >>>>>> do { >>>>>> MPI_Ibarrier(<file’s communicator>, &request); >>>>>> >>>>>> <application stuff> >>>>>> >>>>>> // HDF5 operation: >>>>>> if(<operation is read or write>) { >>>>>> MPI_Wait(&request); >>>>>> <perform read / write> >>>>>> } >>>>>> else { // operation is a file close >>>>>> MPI_Request_free(&request); >>>>>> MPI_File_close(…); >>>>>> MPI_Comm_free(<file’s communicator>); >>>>>> } >>>>>> } while (<file is open>); >>>>>> >>>>>> =============================== >>>>>> >>>>>> What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed of when the nonblocking barrier completes locally. >>>>>> >>>>>> So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request. >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> Quincey >>>>>> >>>>>> >>>>>>> On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hi Jim, >>>>>>> >>>>>>> To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience. >>>>>>> >>>>>>> I am certainly not arguing that it be permitted for more MPI operations. >>>>>>> >>>>>>> I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request. >>>>>>> >>>>>>> If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE without copying it. As you say, MPI takes over the responsibility for processing the completion event. >>>>>>> >>>>>>> Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer to take on this responsibility in the first place?” >>>>>>> >>>>>>> Thanks, that helps (me at least). >>>>>>> >>>>>>> Cheers, >>>>>>> Dan. >>>>>>> — >>>>>>> Dr Daniel Holmes PhD >>>>>>> Architect (HPC Research) >>>>>>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>>>>>> Phone: +44 (0) 131 651 3465 >>>>>>> Mobile: +44 (0) 7940 524 088 >>>>>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>>>>>> — >>>>>>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>>>>>> — >>>>>>> >>>>>>>> On 13 Aug 2020, at 14:43, Jim Dinan <james.dinan(a)gmail.com <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? >>>>>>>> >>>>>>>> My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. >>>>>>>> >>>>>>>> ~Jim. >>>>>>>> >>>>>>>> On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? >>>>>>>> >>>>>>>> MPI_LOCK() ! or any other "open epoch at origin" procedure call >>>>>>>> doUserWorkBefore() >>>>>>>> MPI_RPUT(&req) >>>>>>>> MPI_REQUEST_FREE(&req) >>>>>>>> doUserWorkAfter() >>>>>>>> MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call >>>>>>>> >>>>>>>> vs: >>>>>>>> >>>>>>>> MPI_LOCK() ! or any other "open epoch at origin" procedure call >>>>>>>> doUserWorkBefore() >>>>>>>> MPI_PUT() >>>>>>>> doUserWorkAfter() >>>>>>>> MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call >>>>>>>> >>>>>>>> Is this a source-to-source translation that is always safe in either direction? >>>>>>>> >>>>>>>> In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. >>>>>>>> >>>>>>>> In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. >>>>>>>> In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. >>>>>>>> In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) >>>>>>>> In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. >>>>>>>> >>>>>>>> The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. >>>>>>>> >>>>>>>> Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Dan. >>>>>>>> — >>>>>>>> Dr Daniel Holmes PhD >>>>>>>> Architect (HPC Research) >>>>>>>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>>>>>>> Phone: +44 (0) 131 651 3465 >>>>>>>> Mobile: +44 (0) 7940 524 088 >>>>>>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>>>>>>> — >>>>>>>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>>>>>>> — >>>>>>>> >>>>>>>>> On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types >>>>>>>>> posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after >>>>>>>>> the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. >>>>>>>>> >>>>>>>>> Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). >>>>>>>>> >>>>>>>>> Tony >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Anthony Skjellum, PhD >>>>>>>>> Professor of Computer Science and Chair of Excellence >>>>>>>>> Director, SimCenter >>>>>>>>> University of Tennessee at Chattanooga (UTC) >>>>>>>>> tony-skjellum(a)utc.edu <mailto:[email protected]> [or skjellum(a)gmail.com <mailto:[email protected]>] >>>>>>>>> cell: 205-807-4968 >>>>>>>>> >>>>>>>>> From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> >>>>>>>>> Sent: Saturday, August 8, 2020 12:07 PM >>>>>>>>> To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> >>>>>>>>> Cc: Jeff Hammond <jeff.science(a)gmail.com <mailto:[email protected]>> >>>>>>>>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions >>>>>>>>> >>>>>>>>> We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. >>>>>>>>> >>>>>>>>> Sent from my iPhone >>>>>>>>> >>>>>>>>>> On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>>>>>> >>>>>>>>>> Folks, >>>>>>>>>> >>>>>>>>>> Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. >>>>>>>>>> >>>>>>>>>> Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. >>>>>>>>>> >>>>>>>>>> Can someone clarify or remind me what the reason was? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> — Pavan >>>>>>>>>> >>>>>>>>>> MPI-3.1 standard, page 197, lines 26-27: >>>>>>>>>> >>>>>>>>>> “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> mpi-forum mailing list >>>>>>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>>>>>> _______________________________________________ >>>>>>>>> mpi-forum mailing list >>>>>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>>>>> _______________________________________________ >>>>>>>> mpi-forum mailing list >>>>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi-forum mailing list >>>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi-forum mailing list >>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>> >>>> >>>> _______________________________________________ >>>> mpi-forum mailing list >>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>> >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by HOLMES Daniel 21 Aug '20

21 Aug '20

Hi Quincey, The idea of adding MPI_COMM_IDISCONNECT (also MPI_FILE_ISYNC, as I should have mentioned earlier) is already on the to-do list. Calling MPI_COMM_DISCONNECT replaces the call to MPI_COMM_FREE (both set the comm argument to MPI_COMM_NULL and end the user’s direct interaction/manipulation with/of that communicator). Having called either of these two procedures, the user must not call the other one - and cannot because they no longer have a valid handle for the communicator. That means that MPI_COMM_IDISCONNECT would be somewhat similar to MPI_COMM_FREE (except for the requirement on the user to have already called the completion procedures for pending operations, and it is unlikely to be collective - it will be local, I hope) but, when followed by MPI_WAIT for the request it created as output, it would provide the guarantee given by MPI_COMM_DISCONNECT but not given by MPI_COMM_FREE, i.e. that all pending communication using that communicator has now actually finished internally. Using MPI_TEST permits an entirely nonblocking usage pattern that polls until this guarantee can be made by MPI. Both of these are anticipated to be useful for applications. It would be good to capture your use-case more precisely so that it can be used to justify these changes when they eventually come up for a vote in the MPI Forum. Personally, I believe that MPI_COMM_IDISCONNECT (once it exists) should entirely supplant MPI_COMM_FREE, but that is still somewhat a contentious point-of-view. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 21 Aug 2020, at 16:19, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, Ah, very useful to know, thanks! Is there a nonblocking version of MPI_COMM_DISCONNECT? (I’ve searched the web for MPI_COMM_IDISCONNECT and it comes up empty, but that’s not canonical :-) If not, can a capability like this be added to any “wish lists”? Ideally, calling something like MPI_COMM_IDISCONNECT and then having the request for that operation complete would mean that MPI_COMM_FREE would be guaranteed to be both nonblocking and complete locally. Thoughts? Quincey On Aug 21, 2020, at 10:01 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, Calling MPI_COMM_FREE when some requests representing nonblocking or persistent operations are still active is not prohibited by MPI and seems to work successfully in all the MPI libraries I’ve tested. The normative description for MPI_COMM_FREE in the MPI Standard specifically calls out that it will only mark the communicator for freeing later and may return to the user before pending/ongoing communication is complete. It does not require that the completion procedure has been called for active operations. We discussed in the Forum (as recently as the meeting this week) that this is a key difference between MPI_COMM_FREE and MPI_COMM_DISCONNECT - the latter states that the user is required to call the completion procedure(s) for all operations using a communicator before disconnecting it using MPI_COMM_DISCONNECT, which will wait for all pending communication to complete internally. OTOH, I’m not sure that doing this buys you as much as you think it might. MPI_COMM_FREE is a collective procedure, so it is permitted to wait until MPI_COMM_FREE has been called at all other MPI processes in the communicator, i.e. it can have blocking-barrier-like semantics. All collective operations must be initialised in the same order at all processes in the communicator. So a valid implementation could do all the pending work inside MPI_COMM_FREE but the Standard also permits an implementation that does nothing other than change a “ready-for-freeing” flag on the local communicator object. Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Yes. Will MPI_COMM_FREE block for completion of the NBC op? No. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 21 Aug 2020, at 15:26, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I agree with you about MPI barriers, but that’s why I said it was a simplified pseudocode. :-) We do have more mechanisms in place for handling the "fence-ness” of the operation, but barriers are a component and I’d like to move to a nonblocking version when possible. Having more, hopefully all, of the file operations get equivalent nonblocking versions would be _very_ nice, and I could simplify our internal code more if a nonblocking MPI_FILE_SYNC was available. A nonblocking version of MPI_FILE_SET_SIZE would also be high on my list. Yes, I grok the behavior of MPI_FILE_CLOSE, but don’t want to add a barrier on top of it. :-) One new question: Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Will MPI_COMM_FREE block for completion of the NBC op? Thanks! Quincey On Aug 15, 2020, at 6:07 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files). Perhaps you are looking for MPI_FILE_SYNC? "If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1 "MPI_FILE_SYNC is a collective operation.” §13.6.1 Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally. There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next. As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do: =============================== <open HDF5 file> // sets up a communicator for internal HDF5 communication about this file do { MPI_Ibarrier(<file’s communicator>, &request); <application stuff> // HDF5 operation: if(<operation is read or write>) { MPI_Wait(&request); <perform read / write> } else { // operation is a file close MPI_Request_free(&request); MPI_File_close(…); MPI_Comm_free(<file’s communicator>); } } while (<file is open>); =============================== What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed of when the nonblocking barrier completes locally. So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request. Thoughts? Quincey On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Jim, To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience. I am certainly not arguing that it be permitted for more MPI operations. I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request. If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE without copying it. As you say, MPI takes over the responsibility for processing the completion event. Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer to take on this responsibility in the first place?” Thanks, that helps (me at least). Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 14:43, Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> wrote: The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. ~Jim. On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_RPUT(&req) MPI_REQUEST_FREE(&req) doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call vs: MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_PUT() doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call Is this a source-to-source translation that is always safe in either direction? In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). Tony Anthony Skjellum, PhD Professor of Computer Science and Chair of Excellence Director, SimCenter University of Tennessee at Chattanooga (UTC) tony-skjellum(a)utc.edu<mailto:[email protected]> [or skjellum(a)gmail.com<mailto:[email protected]>] cell: 205-807-4968 ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: Saturday, August 8, 2020 12:07 PM To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. Sent from my iPhone On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Folks, Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. Can someone clarify or remind me what the reason was? Regards, — Pavan MPI-3.1 standard, page 197, lines 26-27: “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Bangalore, Purushotham 21 Aug '20

21 Aug '20

Hi Quincey, There is no MPI_COMM_IDISCONNECT currently, but there is was an effort to add that, you can see the discussion here: https://github.com/mpi-forum/mpi-issues/issues/81 https://github.com/mpi-forum/mpi-issues/issues/78 ==== Puri ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org> Sent: Friday, August 21, 2020 10:19 AM To: MPI Forum <mpi-forum(a)lists.mpi-forum.org> Cc: Quincey Koziol <koziol(a)lbl.gov> Subject: Re: [Mpi-forum] MPI_Request_free restrictions Hi Dan, Ah, very useful to know, thanks! Is there a nonblocking version of MPI_COMM_DISCONNECT? (I’ve searched the web for MPI_COMM_IDISCONNECT and it comes up empty, but that’s not canonical :-) If not, can a capability like this be added to any “wish lists”? Ideally, calling something like MPI_COMM_IDISCONNECT and then having the request for that operation complete would mean that MPI_COMM_FREE would be guaranteed to be both nonblocking and complete locally. Thoughts? Quincey On Aug 21, 2020, at 10:01 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, Calling MPI_COMM_FREE when some requests representing nonblocking or persistent operations are still active is not prohibited by MPI and seems to work successfully in all the MPI libraries I’ve tested. The normative description for MPI_COMM_FREE in the MPI Standard specifically calls out that it will only mark the communicator for freeing later and may return to the user before pending/ongoing communication is complete. It does not require that the completion procedure has been called for active operations. We discussed in the Forum (as recently as the meeting this week) that this is a key difference between MPI_COMM_FREE and MPI_COMM_DISCONNECT - the latter states that the user is required to call the completion procedure(s) for all operations using a communicator before disconnecting it using MPI_COMM_DISCONNECT, which will wait for all pending communication to complete internally. OTOH, I’m not sure that doing this buys you as much as you think it might. MPI_COMM_FREE is a collective procedure, so it is permitted to wait until MPI_COMM_FREE has been called at all other MPI processes in the communicator, i.e. it can have blocking-barrier-like semantics. All collective operations must be initialised in the same order at all processes in the communicator. So a valid implementation could do all the pending work inside MPI_COMM_FREE but the Standard also permits an implementation that does nothing other than change a “ready-for-freeing” flag on the local communicator object. Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Yes. Will MPI_COMM_FREE block for completion of the NBC op? No. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 21 Aug 2020, at 15:26, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I agree with you about MPI barriers, but that’s why I said it was a simplified pseudocode. :-) We do have more mechanisms in place for handling the "fence-ness” of the operation, but barriers are a component and I’d like to move to a nonblocking version when possible. Having more, hopefully all, of the file operations get equivalent nonblocking versions would be _very_ nice, and I could simplify our internal code more if a nonblocking MPI_FILE_SYNC was available. A nonblocking version of MPI_FILE_SET_SIZE would also be high on my list. Yes, I grok the behavior of MPI_FILE_CLOSE, but don’t want to add a barrier on top of it. :-) One new question: Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Will MPI_COMM_FREE block for completion of the NBC op? Thanks! Quincey On Aug 15, 2020, at 6:07 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files). Perhaps you are looking for MPI_FILE_SYNC? "If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1 "MPI_FILE_SYNC is a collective operation.” §13.6.1 Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally. There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next. As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do: =============================== <open HDF5 file> // sets up a communicator for internal HDF5 communication about this file do { MPI_Ibarrier(<file’s communicator>, &request); <application stuff> // HDF5 operation: if(<operation is read or write>) { MPI_Wait(&request); <perform read / write> } else { // operation is a file close MPI_Request_free(&request); MPI_File_close(…); MPI_Comm_free(<file’s communicator>); } } while (<file is open>); =============================== What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed of when the nonblocking barrier completes locally. So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request. Thoughts? Quincey On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Jim, To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience. I am certainly not arguing that it be permitted for more MPI operations. I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request. If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE without copying it. As you say, MPI takes over the responsibility for processing the completion event. Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer to take on this responsibility in the first place?” Thanks, that helps (me at least). Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 14:43, Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> wrote: The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. ~Jim. On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_RPUT(&req) MPI_REQUEST_FREE(&req) doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call vs: MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_PUT() doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call Is this a source-to-source translation that is always safe in either direction? In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). Tony Anthony Skjellum, PhD Professor of Computer Science and Chair of Excellence Director, SimCenter University of Tennessee at Chattanooga (UTC) tony-skjellum(a)utc.edu<mailto:[email protected]> [or skjellum(a)gmail.com<mailto:[email protected]>] cell: 205-807-4968 ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: Saturday, August 8, 2020 12:07 PM To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. Sent from my iPhone On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Folks, Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. Can someone clarify or remind me what the reason was? Regards, — Pavan MPI-3.1 standard, page 197, lines 26-27: “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Quincey Koziol 21 Aug '20

21 Aug '20

Hi Dan, Ah, very useful to know, thanks! Is there a nonblocking version of MPI_COMM_DISCONNECT? (I’ve searched the web for MPI_COMM_IDISCONNECT and it comes up empty, but that’s not canonical :-) If not, can a capability like this be added to any “wish lists”? Ideally, calling something like MPI_COMM_IDISCONNECT and then having the request for that operation complete would mean that MPI_COMM_FREE would be guaranteed to be both nonblocking and complete locally. Thoughts? Quincey > On Aug 21, 2020, at 10:01 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> wrote: > > Hi Quincey, > > Calling MPI_COMM_FREE when some requests representing nonblocking or persistent operations are still active is not prohibited by MPI and seems to work successfully in all the MPI libraries I’ve tested. > > The normative description for MPI_COMM_FREE in the MPI Standard specifically calls out that it will only mark the communicator for freeing later and may return to the user before pending/ongoing communication is complete. It does not require that the completion procedure has been called for active operations. > > We discussed in the Forum (as recently as the meeting this week) that this is a key difference between MPI_COMM_FREE and MPI_COMM_DISCONNECT - the latter states that the user is required to call the completion procedure(s) for all operations using a communicator before disconnecting it using MPI_COMM_DISCONNECT, which will wait for all pending communication to complete internally. > > OTOH, I’m not sure that doing this buys you as much as you think it might. > > MPI_COMM_FREE is a collective procedure, so it is permitted to wait until MPI_COMM_FREE has been called at all other MPI processes in the communicator, i.e. it can have blocking-barrier-like semantics. All collective operations must be initialised in the same order at all processes in the communicator. So a valid implementation could do all the pending work inside MPI_COMM_FREE but the Standard also permits an implementation that does nothing other than change a “ready-for-freeing” flag on the local communicator object. > >> Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? > > Yes. > >> Will MPI_COMM_FREE block for completion of the NBC op? > > No. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Architect (HPC Research) > d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT > — > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > — > >> On 21 Aug 2020, at 15:26, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >> >> Hi Dan, >> I agree with you about MPI barriers, but that’s why I said it was a simplified pseudocode. :-) We do have more mechanisms in place for handling the "fence-ness” of the operation, but barriers are a component and I’d like to move to a nonblocking version when possible. >> >> Having more, hopefully all, of the file operations get equivalent nonblocking versions would be _very_ nice, and I could simplify our internal code more if a nonblocking MPI_FILE_SYNC was available. A nonblocking version of MPI_FILE_SET_SIZE would also be high on my list. >> >> Yes, I grok the behavior of MPI_FILE_CLOSE, but don’t want to add a barrier on top of it. :-) >> >> >> One new question: Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Will MPI_COMM_FREE block for completion of the NBC op? >> >> >> Thanks! >> Quincey >> >> >> >>> On Aug 15, 2020, at 6:07 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]>> wrote: >>> >>> Hi Quincey, >>> >>> The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files). >>> >>> Perhaps you are looking for MPI_FILE_SYNC? >>> >>> "If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1 >>> >>> "MPI_FILE_SYNC is a collective operation.” §13.6.1 >>> >>> Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally. >>> >>> There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next. >>> >>> As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called. >>> >>> Cheers, >>> Dan. >>> — >>> Dr Daniel Holmes PhD >>> Architect (HPC Research) >>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>> Phone: +44 (0) 131 651 3465 >>> Mobile: +44 (0) 7940 524 088 >>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>> — >>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>> — >>> >>>> On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>> >>>> Hi Dan, >>>> I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do: >>>> >>>> =============================== >>>> >>>> <open HDF5 file> // sets up a communicator for internal HDF5 communication about this file >>>> >>>> do { >>>> MPI_Ibarrier(<file’s communicator>, &request); >>>> >>>> <application stuff> >>>> >>>> // HDF5 operation: >>>> if(<operation is read or write>) { >>>> MPI_Wait(&request); >>>> <perform read / write> >>>> } >>>> else { // operation is a file close >>>> MPI_Request_free(&request); >>>> MPI_File_close(…); >>>> MPI_Comm_free(<file’s communicator>); >>>> } >>>> } while (<file is open>); >>>> >>>> =============================== >>>> >>>> What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed of when the nonblocking barrier completes locally. >>>> >>>> So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request. >>>> >>>> Thoughts? >>>> >>>> Quincey >>>> >>>> >>>>> On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>> >>>>> Hi Jim, >>>>> >>>>> To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience. >>>>> >>>>> I am certainly not arguing that it be permitted for more MPI operations. >>>>> >>>>> I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request. >>>>> >>>>> If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE without copying it. As you say, MPI takes over the responsibility for processing the completion event. >>>>> >>>>> Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer to take on this responsibility in the first place?” >>>>> >>>>> Thanks, that helps (me at least). >>>>> >>>>> Cheers, >>>>> Dan. >>>>> — >>>>> Dr Daniel Holmes PhD >>>>> Architect (HPC Research) >>>>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>>>> Phone: +44 (0) 131 651 3465 >>>>> Mobile: +44 (0) 7940 524 088 >>>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>>>> — >>>>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>>>> — >>>>> >>>>>> On 13 Aug 2020, at 14:43, Jim Dinan <james.dinan(a)gmail.com <mailto:[email protected]>> wrote: >>>>>> >>>>>> The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? >>>>>> >>>>>> My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. >>>>>> >>>>>> ~Jim. >>>>>> >>>>>> On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>> Hi all, >>>>>> >>>>>> To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? >>>>>> >>>>>> MPI_LOCK() ! or any other "open epoch at origin" procedure call >>>>>> doUserWorkBefore() >>>>>> MPI_RPUT(&req) >>>>>> MPI_REQUEST_FREE(&req) >>>>>> doUserWorkAfter() >>>>>> MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call >>>>>> >>>>>> vs: >>>>>> >>>>>> MPI_LOCK() ! or any other "open epoch at origin" procedure call >>>>>> doUserWorkBefore() >>>>>> MPI_PUT() >>>>>> doUserWorkAfter() >>>>>> MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call >>>>>> >>>>>> Is this a source-to-source translation that is always safe in either direction? >>>>>> >>>>>> In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. >>>>>> >>>>>> In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. >>>>>> In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. >>>>>> In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) >>>>>> In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. >>>>>> >>>>>> The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. >>>>>> >>>>>> Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. >>>>>> >>>>>> Cheers, >>>>>> Dan. >>>>>> — >>>>>> Dr Daniel Holmes PhD >>>>>> Architect (HPC Research) >>>>>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>>>>> Phone: +44 (0) 131 651 3465 >>>>>> Mobile: +44 (0) 7940 524 088 >>>>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>>>>> — >>>>>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>>>>> — >>>>>> >>>>>>> On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types >>>>>>> posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after >>>>>>> the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. >>>>>>> >>>>>>> Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). >>>>>>> >>>>>>> Tony >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Anthony Skjellum, PhD >>>>>>> Professor of Computer Science and Chair of Excellence >>>>>>> Director, SimCenter >>>>>>> University of Tennessee at Chattanooga (UTC) >>>>>>> tony-skjellum(a)utc.edu <mailto:[email protected]> [or skjellum(a)gmail.com <mailto:[email protected]>] >>>>>>> cell: 205-807-4968 >>>>>>> >>>>>>> From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> >>>>>>> Sent: Saturday, August 8, 2020 12:07 PM >>>>>>> To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> >>>>>>> Cc: Jeff Hammond <jeff.science(a)gmail.com <mailto:[email protected]>> >>>>>>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions >>>>>>> >>>>>>> We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. >>>>>>> >>>>>>> Sent from my iPhone >>>>>>> >>>>>>>> On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Folks, >>>>>>>> >>>>>>>> Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. >>>>>>>> >>>>>>>> Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. >>>>>>>> >>>>>>>> Can someone clarify or remind me what the reason was? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> — Pavan >>>>>>>> >>>>>>>> MPI-3.1 standard, page 197, lines 26-27: >>>>>>>> >>>>>>>> “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi-forum mailing list >>>>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>>>> _______________________________________________ >>>>>>> mpi-forum mailing list >>>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>>> _______________________________________________ >>>>>> mpi-forum mailing list >>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>> >>>>> _______________________________________________ >>>>> mpi-forum mailing list >>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>> >>>> _______________________________________________ >>>> mpi-forum mailing list >>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>> >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by HOLMES Daniel 21 Aug '20

21 Aug '20

Hi Quincey, Calling MPI_COMM_FREE when some requests representing nonblocking or persistent operations are still active is not prohibited by MPI and seems to work successfully in all the MPI libraries I’ve tested. The normative description for MPI_COMM_FREE in the MPI Standard specifically calls out that it will only mark the communicator for freeing later and may return to the user before pending/ongoing communication is complete. It does not require that the completion procedure has been called for active operations. We discussed in the Forum (as recently as the meeting this week) that this is a key difference between MPI_COMM_FREE and MPI_COMM_DISCONNECT - the latter states that the user is required to call the completion procedure(s) for all operations using a communicator before disconnecting it using MPI_COMM_DISCONNECT, which will wait for all pending communication to complete internally. OTOH, I’m not sure that doing this buys you as much as you think it might. MPI_COMM_FREE is a collective procedure, so it is permitted to wait until MPI_COMM_FREE has been called at all other MPI processes in the communicator, i.e. it can have blocking-barrier-like semantics. All collective operations must be initialised in the same order at all processes in the communicator. So a valid implementation could do all the pending work inside MPI_COMM_FREE but the Standard also permits an implementation that does nothing other than change a “ready-for-freeing” flag on the local communicator object. Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Yes. Will MPI_COMM_FREE block for completion of the NBC op? No. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 21 Aug 2020, at 15:26, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I agree with you about MPI barriers, but that’s why I said it was a simplified pseudocode. :-) We do have more mechanisms in place for handling the "fence-ness” of the operation, but barriers are a component and I’d like to move to a nonblocking version when possible. Having more, hopefully all, of the file operations get equivalent nonblocking versions would be _very_ nice, and I could simplify our internal code more if a nonblocking MPI_FILE_SYNC was available. A nonblocking version of MPI_FILE_SET_SIZE would also be high on my list. Yes, I grok the behavior of MPI_FILE_CLOSE, but don’t want to add a barrier on top of it. :-) One new question: Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Will MPI_COMM_FREE block for completion of the NBC op? Thanks! Quincey On Aug 15, 2020, at 6:07 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]>> wrote: Hi Quincey, The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files). Perhaps you are looking for MPI_FILE_SYNC? "If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1 "MPI_FILE_SYNC is a collective operation.” §13.6.1 Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally. There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next. As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dan, I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do: =============================== <open HDF5 file> // sets up a communicator for internal HDF5 communication about this file do { MPI_Ibarrier(<file’s communicator>, &request); <application stuff> // HDF5 operation: if(<operation is read or write>) { MPI_Wait(&request); <perform read / write> } else { // operation is a file close MPI_Request_free(&request); MPI_File_close(…); MPI_Comm_free(<file’s communicator>); } } while (<file is open>); =============================== What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed of when the nonblocking barrier completes locally. So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request. Thoughts? Quincey On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Jim, To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience. I am certainly not arguing that it be permitted for more MPI operations. I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request. If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE without copying it. As you say, MPI takes over the responsibility for processing the completion event. Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer to take on this responsibility in the first place?” Thanks, that helps (me at least). Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 14:43, Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> wrote: The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. ~Jim. On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_RPUT(&req) MPI_REQUEST_FREE(&req) doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call vs: MPI_LOCK() ! or any other "open epoch at origin" procedure call doUserWorkBefore() MPI_PUT() doUserWorkAfter() MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call Is this a source-to-source translation that is always safe in either direction? In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). Tony Anthony Skjellum, PhD Professor of Computer Science and Chair of Excellence Director, SimCenter University of Tennessee at Chattanooga (UTC) tony-skjellum(a)utc.edu<mailto:[email protected]> [or skjellum(a)gmail.com<mailto:[email protected]>] cell: 205-807-4968 ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: Saturday, August 8, 2020 12:07 PM To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. Sent from my iPhone On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Folks, Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. Can someone clarify or remind me what the reason was? Regards, — Pavan MPI-3.1 standard, page 197, lines 26-27: “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Request_free restrictions
by Quincey Koziol 21 Aug '20

21 Aug '20

Hi Dan, I agree with you about MPI barriers, but that’s why I said it was a simplified pseudocode. :-) We do have more mechanisms in place for handling the "fence-ness” of the operation, but barriers are a component and I’d like to move to a nonblocking version when possible. Having more, hopefully all, of the file operations get equivalent nonblocking versions would be _very_ nice, and I could simplify our internal code more if a nonblocking MPI_FILE_SYNC was available. A nonblocking version of MPI_FILE_SET_SIZE would also be high on my list. Yes, I grok the behavior of MPI_FILE_CLOSE, but don’t want to add a barrier on top of it. :-) One new question: Am I allowed to call MPI_COMM_FREE while I have an uncompleted request for a nonblocking collective operation (like MPI_IBARRIER) on the communicator? Will MPI_COMM_FREE block for completion of the NBC op? Thanks! Quincey > On Aug 15, 2020, at 6:07 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> wrote: > > Hi Quincey, > > The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files). > > Perhaps you are looking for MPI_FILE_SYNC? > > "If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1 > > "MPI_FILE_SYNC is a collective operation.” §13.6.1 > > Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally. > > There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next. > > As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Architect (HPC Research) > d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT > — > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > — > >> On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >> >> Hi Dan, >> I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do: >> >> =============================== >> >> <open HDF5 file> // sets up a communicator for internal HDF5 communication about this file >> >> do { >> MPI_Ibarrier(<file’s communicator>, &request); >> >> <application stuff> >> >> // HDF5 operation: >> if(<operation is read or write>) { >> MPI_Wait(&request); >> <perform read / write> >> } >> else { // operation is a file close >> MPI_Request_free(&request); >> MPI_File_close(…); >> MPI_Comm_free(<file’s communicator>); >> } >> } while (<file is open>); >> >> =============================== >> >> What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed of when the nonblocking barrier completes locally. >> >> So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request. >> >> Thoughts? >> >> Quincey >> >> >>> On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>> >>> Hi Jim, >>> >>> To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience. >>> >>> I am certainly not arguing that it be permitted for more MPI operations. >>> >>> I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request. >>> >>> If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE without copying it. As you say, MPI takes over the responsibility for processing the completion event. >>> >>> Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer to take on this responsibility in the first place?” >>> >>> Thanks, that helps (me at least). >>> >>> Cheers, >>> Dan. >>> — >>> Dr Daniel Holmes PhD >>> Architect (HPC Research) >>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>> Phone: +44 (0) 131 651 3465 >>> Mobile: +44 (0) 7940 524 088 >>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>> — >>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>> — >>> >>>> On 13 Aug 2020, at 14:43, Jim Dinan <james.dinan(a)gmail.com <mailto:[email protected]>> wrote: >>>> >>>> The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should we ask implementers to take on this complexity and overhead? >>>> >>>> My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput. I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to meet MPI's quiescent shutdown requirement. >>>> >>>> ~Jim. >>>> >>>> On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>> Hi all, >>>> >>>> To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation? >>>> >>>> MPI_LOCK() ! or any other "open epoch at origin" procedure call >>>> doUserWorkBefore() >>>> MPI_RPUT(&req) >>>> MPI_REQUEST_FREE(&req) >>>> doUserWorkAfter() >>>> MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call >>>> >>>> vs: >>>> >>>> MPI_LOCK() ! or any other "open epoch at origin" procedure call >>>> doUserWorkBefore() >>>> MPI_PUT() >>>> doUserWorkAfter() >>>> MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call >>>> >>>> Is this a source-to-source translation that is always safe in either direction? >>>> >>>> In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation. >>>> >>>> In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives. >>>> In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point. >>>> In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.) >>>> In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called. >>>> >>>> The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules. >>>> >>>> Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure. >>>> >>>> Cheers, >>>> Dan. >>>> — >>>> Dr Daniel Holmes PhD >>>> Architect (HPC Research) >>>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>>> Phone: +44 (0) 131 651 3465 >>>> Mobile: +44 (0) 7940 524 088 >>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >>>> — >>>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>>> — >>>> >>>>> On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>> >>>>> FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... The MPI_Request_free on an NBC poses a problem for the cases where there are array types >>>>> posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after >>>>> the free on an active request. We do *not* mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI implementation till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. >>>>> >>>>> Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize). >>>>> >>>>> Tony >>>>> >>>>> >>>>> >>>>> >>>>> Anthony Skjellum, PhD >>>>> Professor of Computer Science and Chair of Excellence >>>>> Director, SimCenter >>>>> University of Tennessee at Chattanooga (UTC) >>>>> tony-skjellum(a)utc.edu <mailto:[email protected]> [or skjellum(a)gmail.com <mailto:[email protected]>] >>>>> cell: 205-807-4968 >>>>> >>>>> From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> >>>>> Sent: Saturday, August 8, 2020 12:07 PM >>>>> To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> >>>>> Cc: Jeff Hammond <jeff.science(a)gmail.com <mailto:[email protected]>> >>>>> Subject: Re: [Mpi-forum] MPI_Request_free restrictions >>>>> >>>>> We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >>>>>> >>>>>> Folks, >>>>>> >>>>>> Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC requests so they don’t have to wait for the operation to complete in some situations. >>>>>> >>>>>> Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now. >>>>>> >>>>>> Can someone clarify or remind me what the reason was? >>>>>> >>>>>> Regards, >>>>>> >>>>>> — Pavan >>>>>> >>>>>> MPI-3.1 standard, page 197, lines 26-27: >>>>>> >>>>>> “It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL for a request associated with a nonblocking collective operation.” >>>>>> >>>>>> _______________________________________________ >>>>>> mpi-forum mailing list >>>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>>> _______________________________________________ >>>>> mpi-forum mailing list >>>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>>> _______________________________________________ >>>> mpi-forum mailing list >>>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >>> >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Re: [Mpi-forum] Final Reminder: Deadline for Ballots, Readings, ... for the August MPI Forum Meeting is Tomorrow/Monday
by Rolf Rabenseifner 17 Aug '20

17 Aug '20

Dear all, I updated the issue based on the discussion today: https://github.com/mpi-forum/mpi-issues/files/5086553/mpi-report-issue120-t… contains the change for the no-no-vote tomorrow. Best regards and thanks to all for this thorough review, Rolf ----- Original Message ----- > From: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Cc: "Guillaume Mercier" <guillaume.mercier(a)u-bordeaux.fr> > Sent: Monday, August 3, 2020 6:39:24 PM > Subject: Re: [Mpi-forum] Final Reminder: Deadline for Ballots, Readings, ... for the August MPI Forum Meeting is > Tomorrow/Monday > Hi all, > > The Hardware Topologies Working Group wants to announce the first vote > for the MPI_Cart_create_weighted / Topology aware Cartesian communicators: > > Annotated pdf as read in the June 2020 meeting: > https://github.com/mpi-forum/mpi-issues/files/4823780/mpi-report-issue120-t… > > Issue https://github.com/mpi-forum/mpi-issues/issues/120 > PR#98 https://github.com/mpi-forum/mpi-standard/pull/98 > > This is one of the tickets for MPI-4.0 for performance enhancements. > > It fills the gap that the MPI-1.1 MPI_Dims_create wasn't hardware nor > application topology aware. > > For this, it combines both functionalities, > - the factorization of #processes into the given number of dimensions > (as non-hardware/application aware in MPI_Dims_create) > - the reordering of the processes and providing comm_cart > (as in MPI_Cart_create) > into one new hardware & application aware routine: MPI_Cart_create_weighted. > > The hardware awareness comes through the old_comm, > the application awareness through the weights. > > > The design of the interface was developed with the help of many forum > members (thanks a lot!) and is along the principles we already use for > the weights for graph topologies for application topology awareness. > This also means, there is enough room for further research and > development through an info argument to allow additional features in the > future or for specific vendor platforms. > > The interface is as simple as possible according to the main goal of > MPI as defined on page 1 of all MPI standards: > > "The goal of the Message-Passing Interface simply stated is to > develop a widely used standard for writing message-passing programs. > As such the interface should establish a practical, portable, > efficient, and flexible standard for message passing." > > For Cartesian applications, the new interface fulfills this short list > of being "practical, portable, efficient, and flexible" and can be > widely used for all applications that use already the old MPI-1.1 > interface MPI_Dims_create + MPI_Cart_create to realize better > hardware-awareness and to allow also application-awareness through > the new weights. > > And for portable development of MPI applications, it is equally > important that the new interface is part of the MPI standard as it > was for the old Cartesian interface. In addition, best optimization > may require knowledge of the hardware which may not be disclosed > to the public which would prevent third party solutions - this serves > as second argument to vote the new interface into the MPI library. > > Therefore, we'll be glad if you vote for this new performance > oriented interface for MPI-4.0. > > And thank you for reading this announcement to the end. > > Best regards, > Guillaume and the Hardware Topologies Working Group > > > > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Reminder / Agenda - MPI Forum Virtual Voting Meeting starting TODAY (Monday, 9am CDT)
by Martin Schulz 17 Aug '20

17 Aug '20

Hi all, Just as a last minute reminder - the virtual voting MPI Forum meeting starts today at 9am CDT and are scheduling to run until noon - same for the next two days (assuming we need the entire time). The agenda (incl. the timing of the voting blocks) is here: https://www.mpi-forum.org/meetings/2020/08/agenda We'll start (after the usual intro) with the NoNo and Errata Readings and whatever we get through, we will vote on today. All other items, plus the scheduled first and second votes, will be in the voting block tomorrow. If you haven't registered, yet, please see Wesley's previous email! If we missed something on the agenda or something new came up, please let us know. Talk to you in a short while, Thanks! Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de

1 0