On Tue, Oct 30, 2012 at 09:09:15AM -0500, Pavan Balaji wrote:

On 10/30/2012 08:26 AM, Torsten Hoefler wrote:

...

...

Actually, the flush only needs to block till its visible to another MPI RMA operation, not to the CPU (as in, not to load/store). In the unified memory model, it has to guarantee visibility to load/store operations as well.

How will the origin process guarantee this without a WIN_SYNC?

Well, this depends on the architecture. On x86, for example, the origin could issue an mfence which would guarantee global ordering (sequential consistency for this statement). The "hand waiving" (barrier or whatnot) would then be ordered after that and cannot pass the store at the target.

...

...

I don't think the target process can guarantee that the PUT is visible to a load/store without an additional memory barrier. The flush of the source process has to ensure that.

How? See above, mfence.

...

...

Since in the MPI standard, we don't specify that the user needs to call a WIN_SYNC in this case, I'm asking if the MPI implementation needs to do a memory barrier internally. Yes, win_synch is only needed in the separate model, which would then do a memory barrier. The MPI library has to guarantee that there is no inconsistency between public and private copy in the unified model (or it cannot claim that it supports unified).

I'm not sure any architecture can do this without internally adding memory barriers in a lot of MPI operations other than the usual FLUSH/SYNC/UNLOCK type of operations. I think this would be possible, given the loose consistency we specify.

Note that the MPI standard only says that in the UNIFIED model, the data will "eventually" appear (pg. 454, bullet item 6). No guarantees that the data actually is visible for load operations immediately after a flush. Yes, I realized that while trying to formalize the memory model. This contradicts other statements like "public and private copies are identical" (p. 436) in conjunction with "the operation is completed at the target [by a call to win_flush]" (p. 454). I think we should reconsider those "eventual" clauses because, if we assume that there is no happens-before consistency guarantee, and instead just a weak "in some time" statement, then we could never use the unified model for anything correct (because "eventually" may as well be in 10^16 years). While, if we remove this eventual statements, the semantics make sense and seem implementable.

I have this on a longer list of issues to bring up at the next forum meeting. Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Z�rich Universit�tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29