On Tue, Oct 30, 2012 at 10:44:06AM -0500, Pavan Balaji wrote:

On 10/30/2012 10:13 AM, Jeff Hammond wrote:

...

...

...

Even in shared memory, the mfence is useless for guaranteeing visibility to any other thread/process. For visibility on architectures that reorder loads, the P1 must issue a read memory barrier after seeing the hand wave and before reading from the window.

Right, so it's possible that on platforms that require a read barrier because they reorder loads, the MPI implementation will not be able to support the unified model. Or they'll have to have a read barrier before every get in the unified model or some such thing.

Seems perfectly reasonable to have a read barrier before a Get on a processor with a weaker memory model.

That can be expensive to do this for every local GET operation.

Well, some architectures (x86) just enable this by default and take the cost of the missed opportunities (mostly speculation afaik) for each memory operation. This may now turn into an endless architecture debate :-).

But I think we are converging --

for shared memory, we do agree that you need some sort of memory barrier within the GET operation for all local GETs. I agree that on x86-like architectures, this might not be needed. Yes

for network operations, this is harder unless the network guarantees that the data is placed in the remote memory. This, for example, is guaranteed by Mellanox adapters, but not by InfiniBand itself. Yes, I always assumed IB guarantees that but I couldn't cite the page of the standard. Isn't it that for an RDMA, the CQ entry will only appear at the source if the message is committed to remote memory?

Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Z�rich Universit�tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29