October 2012 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Jeff Hammond 30 Oct '12

30 Oct '12

>> Actually, the flush only needs to block till its visible to another MPI >> RMA operation, not to the CPU (as in, not to load/store). > In the unified memory model, it has to guarantee visibility to > load/store operations as well. MPI-3 11.4 "In the RMA unified model, public and private copies are identical and updates via put or accumulate calls are eventually observed by load operations without additional RMA calls. A store access to a window is eventually visible to remote get or accumulate calls without additional RMA calls. These stronger semantics of the RMA unified model allow the user to omit some synchronization calls and potentially improve performance." The important word here is _eventually_. The standard does not say _immediately_. In both the separate and unified models, remote completion via e.g. unlock ensures global visibility to all RMA operations _immediately_. On the other hand, without an explicit RMA call, one has to assume that visibility to load instructions is _not_ immediate and therefore requires some additional synchronization mechanism is immediacy is required. If you guys wanted RMA ops to ensure immediate visibility to load instructions, you should have put that in the standard :-) Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Torsten Hoefler 30 Oct '12

30 Oct '12

On Tue, Oct 30, 2012 at 08:00:12AM -0700, Jed Brown wrote: > On Tue, Oct 30, 2012 at 6:54 AM, Torsten Hoefler <[1]htor(a)illinois.edu> > wrote: > > >>> I don't think ï¿½the target process can guarantee that the PUT is > >>> visible to a load/store ï¿½without an additional memory barrier. > >> The flush of the source process has to ensure that. > > > > How? > See above, mfence. > > Even in shared memory, the mfence is useless for guaranteeing visibility > to any other thread/process. For visibility on architectures that reorder > loads, the P1 must issue a read memory barrier after seeing the hand wave > and before reading from the window. My stated assumption was TSO/x86, which does not reorder loads. You are of course correct in the general case (ARM for example). Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Underwood, Keith D 30 Oct '12

30 Oct '12

> > Ok, but, the kind of thing that delays that visibility (e.g. the > > uncertainty between when a posted write starts over PCIExpress and > > when it is known to be globally visible) is completely unaffected by a > > memory barrier. Memory barrier until your heart is content and that > > thing could still be getting retried across a PCIExpress link and the > > process (and the NIC and other side of the link and just about > > anything else) would have no way to know... > > Hmm. Yes, you are right. That's even worse though :-(. Not usually. If you start synchronizing outside the network you better know exactly what you are doing. Most networks have some ordering primitives to help here.

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Jed Brown 30 Oct '12

30 Oct '12

On Tue, Oct 30, 2012 at 6:26 AM, Torsten Hoefler <htor(a)illinois.edu> wrote: > > Since in the MPI standard, we don't specify that the user needs to > > call a WIN_SYNC in this case, I'm asking if the MPI implementation > > needs to do a memory barrier internally. > Yes, win_synch is only needed in the separate model, which would then do > a memory barrier. The MPI library has to guarantee that there is no > inconsistency between public and private copy in the unified model (or > it cannot claim that it supports unified). I thought we had gone through this example before in the thread with Maclaren, but in general, the reader needs to issue a read memory barrier after seeing the hand wave and before accessing the window. There is nothing that the writer can do to guarantee memory ordering for ensuing the local read at the target (this could be happening in a threaded environment). Since x86 does not reorder loads, you don't have to do anything there, but on POWER and ARM (and of course the dreaded Alpha), you need the read memory barrier.

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Pavan Balaji 30 Oct '12

30 Oct '12

On 10/30/2012 09:13 AM, Underwood, Keith D wrote: >>> Yes, win_synch is only needed in the separate model, which would then >>> do a memory barrier. The MPI library has to guarantee that there is no >>> inconsistency between public and private copy in the unified model (or >>> it cannot claim that it supports unified). >> >> I'm not sure any architecture can do this without internally adding memory >> barriers in a lot of MPI operations other than the usual FLUSH/SYNC/UNLOCK >> type of operations. >> >> Note that the MPI standard only says that in the UNIFIED model, the data will >> "eventually" appear (pg. 454, bullet item 6). No guarantees that the data >> actually is visible for load operations immediately after a flush. > > Ok, but, the kind of thing that delays that visibility (e.g. the > uncertainty between when a posted write starts over PCIExpress and > when it is known to be globally visible) is completely unaffected by > a memory barrier. Memory barrier until your heart is content and > that thing could still be getting retried across a PCIExpress link > and the process (and the NIC and other side of the link and just > about anything else) would have no way to know... Hmm. Yes, you are right. That's even worse though :-(. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Underwood, Keith D 30 Oct '12

30 Oct '12

> > Yes, win_synch is only needed in the separate model, which would then > > do a memory barrier. The MPI library has to guarantee that there is no > > inconsistency between public and private copy in the unified model (or > > it cannot claim that it supports unified). > > I'm not sure any architecture can do this without internally adding memory > barriers in a lot of MPI operations other than the usual FLUSH/SYNC/UNLOCK > type of operations. > > Note that the MPI standard only says that in the UNIFIED model, the data will > "eventually" appear (pg. 454, bullet item 6). No guarantees that the data > actually is visible for load operations immediately after a flush. Ok, but, the kind of thing that delays that visibility (e.g. the uncertainty between when a posted write starts over PCIExpress and when it is known to be globally visible) is completely unaffected by a memory barrier. Memory barrier until your heart is content and that thing could still be getting retried across a PCIExpress link and the process (and the NIC and other side of the link and just about anything else) would have no way to know... Keith

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Pavan Balaji 30 Oct '12

30 Oct '12

On 10/30/2012 08:26 AM, Torsten Hoefler wrote: >> Actually, the flush only needs to block till its visible to another MPI >> RMA operation, not to the CPU (as in, not to load/store). > In the unified memory model, it has to guarantee visibility to > load/store operations as well. How will the origin process guarantee this without a WIN_SYNC? >> I don't think the target process can guarantee that the PUT is >> visible to a load/store without an additional memory barrier. > The flush of the source process has to ensure that. How? >> Since in the MPI standard, we don't specify that the user needs to >> call a WIN_SYNC in this case, I'm asking if the MPI implementation >> needs to do a memory barrier internally. > Yes, win_synch is only needed in the separate model, which would then do > a memory barrier. The MPI library has to guarantee that there is no > inconsistency between public and private copy in the unified model (or > it cannot claim that it supports unified). I'm not sure any architecture can do this without internally adding memory barriers in a lot of MPI operations other than the usual FLUSH/SYNC/UNLOCK type of operations. Note that the MPI standard only says that in the UNIFIED model, the data will "eventually" appear (pg. 454, bullet item 6). No guarantees that the data actually is visible for load operations immediately after a flush. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Torsten Hoefler 30 Oct '12

30 Oct '12

On Tue, Oct 30, 2012 at 02:21:12PM +0000, Underwood, Keith D wrote: > > > Ok, but, the kind of thing that delays that visibility (e.g. the > > > uncertainty between when a posted write starts over PCIExpress and > > > when it is known to be globally visible) is completely unaffected by a > > > memory barrier. Memory barrier until your heart is content and that > > > thing could still be getting retried across a PCIExpress link and the > > > process (and the NIC and other side of the link and just about > > > anything else) would have no way to know... > > > > Hmm. Yes, you are right. That's even worse though :-(. > > Not usually. If you start synchronizing outside the network you > better know exactly what you are doing. Most networks have some > ordering primitives to help here. Yes, so I assumed that this was all happening in shared memory (thus the mfence). If we're on a network, we need to ensure that we have ordering between the fence and the hand waiving :-). As for the retry, I think the flush may not return before the value is successfully committed into the target memory (either public or private copy). Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Torsten Hoefler 30 Oct '12

30 Oct '12

On Tue, Oct 30, 2012 at 09:09:15AM -0500, Pavan Balaji wrote: > > On 10/30/2012 08:26 AM, Torsten Hoefler wrote: >>> Actually, the flush only needs to block till its visible to another MPI >>> RMA operation, not to the CPU (as in, not to load/store). >> In the unified memory model, it has to guarantee visibility to >> load/store operations as well. > > How will the origin process guarantee this without a WIN_SYNC? Well, this depends on the architecture. On x86, for example, the origin could issue an mfence which would guarantee global ordering (sequential consistency for this statement). The "hand waiving" (barrier or whatnot) would then be ordered after that and cannot pass the store at the target. >>> I don't think the target process can guarantee that the PUT is >>> visible to a load/store without an additional memory barrier. >> The flush of the source process has to ensure that. > > How? See above, mfence. >>> Since in the MPI standard, we don't specify that the user needs to >>> call a WIN_SYNC in this case, I'm asking if the MPI implementation >>> needs to do a memory barrier internally. >> Yes, win_synch is only needed in the separate model, which would then do >> a memory barrier. The MPI library has to guarantee that there is no >> inconsistency between public and private copy in the unified model (or >> it cannot claim that it supports unified). > > I'm not sure any architecture can do this without internally adding > memory barriers in a lot of MPI operations other than the usual > FLUSH/SYNC/UNLOCK type of operations. I think this would be possible, given the loose consistency we specify. > Note that the MPI standard only says that in the UNIFIED model, the data > will "eventually" appear (pg. 454, bullet item 6). No guarantees that > the data actually is visible for load operations immediately after a > flush. Yes, I realized that while trying to formalize the memory model. This contradicts other statements like "public and private copies are identical" (p. 436) in conjunction with "the operation is completed at the target [by a call to win_flush]" (p. 454). I think we should reconsider those "eventual" clauses because, if we assume that there is no happens-before consistency guarantee, and instead just a weak "in some time" statement, then we could never use the unified model for anything correct (because "eventually" may as well be in 10^16 years). While, if we remove this eventual statements, the semantics make sense and seem implementable. I have this on a longer list of issues to bring up at the next forum meeting. Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Pavan Balaji 30 Oct '12

30 Oct '12

On 10/30/2012 07:28 AM, Torsten Hoefler wrote: >> If the MPI_GET on P1 does a local load operation internally, it is not >> memory consistent without a memory barrier. Does this mean that I need >> to always do a memory barrier on all local GET operations to get the >> right value? > Well, if the put came from shared memory, then the flush would most > likely do a memory barrier (depending on the consistency model of the > architecture, i.e., in TSO (x86) it would need an mfence). If the put > came from remote memory then there should not be a problem because the > flush must block until the data is visible to the CPU (assuming the > unified memory model, if you were in separate, you'd need an additional > win_synch). Actually, the flush only needs to block till its visible to another MPI RMA operation, not to the CPU (as in, not to load/store). I don't think the target process can guarantee that the PUT is visible to a load/store without an additional memory barrier. Since in the MPI standard, we don't specify that the user needs to call a WIN_SYNC in this case, I'm asking if the MPI implementation needs to do a memory barrier internally. >> Note that this is likely only a theoretical exercise, since most (all?) >> compilers will do a memory barrier anyway if they see a function call >> (MPI_GET in this case). But is MPI assuming that that's going to be the >> case for efficient execution? > Really? Which compiler puts a memory barrier before function calls? This > sounds rather inefficient to me (in the days of fastcall). Sorry, I meant it'll not reorder operations across function calls, rather than a memory barrier. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0