October 2012 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Pavan Balaji 30 Oct '12

30 Oct '12

On 10/30/2012 10:50 AM, Underwood, Keith D wrote: >> for shared memory, we do agree that you need some sort of memory >> barrier within the GET operation for all local GETs. I agree that >> on x86-like architectures, this might not be needed. > > Why is it in the GET instead of the sync that precedes it? Without > the sync, the memory barrier in the GET does not have value, does > it? My point with the example I gave is that the synchronization might not be an RMA Win_sync. In my example, it was the hand waving. If we require the user to do MPI_WIN_SYNC before doing an MPI_GET from local windows, the shared memory problem will be resolved. However, as you pointed out, this still doesn't resolve the network problem unless the FLUSH guarantees that the data is available in remote memory and not stuck in the PCIe complex somewhere. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Pavan Balaji 30 Oct '12

30 Oct '12

On 10/30/2012 10:17 AM, Torsten Hoefler wrote: >> for network operations, this is harder unless the network guarantees >> that the data is placed in the remote memory. This, for example, is >> guaranteed by Mellanox adapters, but not by InfiniBand itself. > Yes, I always assumed IB guarantees that but I couldn't cite the page > of the standard. Isn't it that for an RDMA, the CQ entry will only > appear at the source if the message is committed to remote memory? No, it just means that it has been reliably delivered to the remote network adapter. The Mellanox adapters, however, send the network hardware ACK only after they get the PCIe ACK for the memory commit. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Underwood, Keith D 30 Oct '12

30 Oct '12

> for shared memory, we do agree that you need some sort of memory barrier > within the GET operation for all local GETs. I agree that on x86-like > architectures, this might not be needed. Why is it in the GET instead of the sync that precedes it? Without the sync, the memory barrier in the GET does not have value, does it?

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Underwood, Keith D 30 Oct '12

30 Oct '12

> >>Even in shared memory, the mfence is useless for guaranteeing > >>visibility to any other thread/process. For visibility on > >>architectures that reorder loads, the P1 must issue a read memory > >>barrier after seeing the hand wave and before reading from the window. > > > > Right, so it's possible that on platforms that require a read barrier > > because they reorder loads, the MPI implementation will not be able to > > support the unified model. Or they'll have to have a read barrier > > before every get in the unified model or some such thing. > > Seems perfectly reasonable to have a read barrier before a Get on a > processor with a weaker memory model. But, that read barrier should be in the magic synchronization and not in the MPI_GET, shouldn't it? In a unified memory model, you only have that synchronization to know WHEN the thing will be visible. This should only be a reordering issue. Any synchronization that allows a load to migrate across it - whether due to the compiler or the architecture - is not a synchronization at all.

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Pavan Balaji 30 Oct '12

30 Oct '12

On 10/30/2012 10:13 AM, Jeff Hammond wrote: >>> Even in shared memory, the mfence is useless for guaranteeing visibility >>> to any other thread/process. For visibility on architectures that reorder >>> loads, the P1 must issue a read memory barrier after seeing the hand wave >>> and before reading from the window. >> >> Right, so it's possible that on platforms that require a read barrier >> because they reorder loads, the MPI implementation will not be able to >> support the unified model. Or they'll have to have a read barrier before >> every get in the unified model or some such thing. > > Seems perfectly reasonable to have a read barrier before a Get on a > processor with a weaker memory model. That can be expensive to do this for every local GET operation. But I think we are converging -- for shared memory, we do agree that you need some sort of memory barrier within the GET operation for all local GETs. I agree that on x86-like architectures, this might not be needed. for network operations, this is harder unless the network guarantees that the data is placed in the remote memory. This, for example, is guaranteed by Mellanox adapters, but not by InfiniBand itself. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Torsten Hoefler 30 Oct '12

30 Oct '12

On Tue, Oct 30, 2012 at 10:44:06AM -0500, Pavan Balaji wrote: > > On 10/30/2012 10:13 AM, Jeff Hammond wrote: >>>> Even in shared memory, the mfence is useless for guaranteeing visibility >>>> to any other thread/process. For visibility on architectures that reorder >>>> loads, the P1 must issue a read memory barrier after seeing the hand wave >>>> and before reading from the window. >>> >>> Right, so it's possible that on platforms that require a read barrier >>> because they reorder loads, the MPI implementation will not be able to >>> support the unified model. Or they'll have to have a read barrier before >>> every get in the unified model or some such thing. >> >> Seems perfectly reasonable to have a read barrier before a Get on a >> processor with a weaker memory model. > > That can be expensive to do this for every local GET operation. Well, some architectures (x86) just enable this by default and take the cost of the missed opportunities (mostly speculation afaik) for each memory operation. This may now turn into an endless architecture debate :-). > But I think we are converging -- > > for shared memory, we do agree that you need some sort of memory barrier > within the GET operation for all local GETs. I agree that on x86-like > architectures, this might not be needed. Yes > for network operations, this is harder unless the network guarantees > that the data is placed in the remote memory. This, for example, is > guaranteed by Mellanox adapters, but not by InfiniBand itself. Yes, I always assumed IB guarantees that but I couldn't cite the page of the standard. Isn't it that for an RDMA, the CQ entry will only appear at the source if the message is committed to remote memory? Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Jeff Hammond 30 Oct '12

30 Oct '12

>>Even in shared memory, the mfence is useless for guaranteeing visibility >>to any other thread/process. For visibility on architectures that reorder >>loads, the P1 must issue a read memory barrier after seeing the hand wave >>and before reading from the window. > > Right, so it's possible that on platforms that require a read barrier > because they reorder loads, the MPI implementation will not be able to > support the unified model. Or they'll have to have a read barrier before > every get in the unified model or some such thing. Seems perfectly reasonable to have a read barrier before a Get on a processor with a weaker memory model. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Jed Brown 30 Oct '12

30 Oct '12

On Tue, Oct 30, 2012 at 8:07 AM, Barrett, Brian W <bwbarre(a)sandia.gov>wrote: > On 10/30/12 9:00 AM, "Jed Brown" <jedbrown(a)mcs.anl.gov> wrote: > > >On Tue, Oct 30, 2012 at 6:54 AM, Torsten Hoefler <htor(a)illinois.edu> > >wrote: > > > >>>> I don't think the target process can guarantee that the PUT is > >>>> visible to a load/store without an additional memory barrier. > >>> The flush of the source process has to ensure that. > >> > >> How? > > > >See above, mfence. > > > >Even in shared memory, the mfence is useless for guaranteeing visibility > >to any other thread/process. For visibility on architectures that reorder > >loads, the P1 must issue a read memory barrier after seeing the hand wave > >and before reading from the window. > > Right, so it's possible that on platforms that require a read barrier > because they reorder loads, the MPI implementation will not be able to > support the unified model. Or they'll have to have a read barrier before > every get in the unified model or some such thing. Right, I think it would be a cop-out to say that UNIFIED would not be supported there, thus the read memory barrier.

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Memory barriers in MPI_WIN_LOCK_ALL mode
by Barrett, Brian W 30 Oct '12

30 Oct '12

On 10/30/12 9:00 AM, "Jed Brown" <jedbrown(a)mcs.anl.gov> wrote: >On Tue, Oct 30, 2012 at 6:54 AM, Torsten Hoefler <htor(a)illinois.edu> >wrote: > >>>> I don't think the target process can guarantee that the PUT is >>>> visible to a load/store without an additional memory barrier. >>> The flush of the source process has to ensure that. >> >> How? > >See above, mfence. > >Even in shared memory, the mfence is useless for guaranteeing visibility >to any other thread/process. For visibility on architectures that reorder >loads, the P1 must issue a read memory barrier after seeing the hand wave >and before reading from the window. Right, so it's possible that on platforms that require a read barrier because they reorder loads, the MPI implementation will not be able to support the unified model. Or they'll have to have a read barrier before every get in the unified model or some such thing. Brian

1 0

Re: [Mpi3-rma] Memory barriers in MPI_WIN_LOCK_ALL mode
by Jed Brown 30 Oct '12

30 Oct '12

On Tue, Oct 30, 2012 at 6:54 AM, Torsten Hoefler <htor(a)illinois.edu> wrote: > >>> I don't think the target process can guarantee that the PUT is > >>> visible to a load/store without an additional memory barrier. > >> The flush of the source process has to ensure that. > > > > How? > See above, mfence. Even in shared memory, the mfence is useless for guaranteeing visibility to any other thread/process. For visibility on architectures that reorder loads, the P1 must issue a read memory barrier after seeing the hand wave and before reading from the window.

1 0