Brian, Pavan, and Jeff, you convinced me. I did it, see attached file, and my mpich based Cray lib tells Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in MPI_Win_sync: Wrong synchronization of RMA calls , error stack: MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed MPIDI_Win_sync(2495): Wrong synchronization of RMA calls (only once in each process). I expect, that this is now an implementation bug that should be fixed by mpich and cray? Best regards Rolf ----- Original Message -----
From: "Brian W Barrett" <[email protected]> To: "MPI WG Remote Memory Access working group" <[email protected]> Cc: "Stefan Andersson" <[email protected]>, "Bill Long" <[email protected]> Sent: Tuesday, February 4, 2014 7:09:02 PM Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
On 2/4/14 11:01 AM, "Rolf Rabenseifner" <[email protected]> wrote:
The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) has no meaning in the unified memory model if all accesses are done without RMA routines. It has only a meaning if different public and privat copy is there (MPI-3.0 p450:46-p451:2). MPI-3.0 p456:3 - p457:7 define the rules for the unified memory model but there is no need to use MPI_WIN_SYNC.
Right, there's no need from an MPI point of view, but that doesn't mean that the language/compiler/processor doesn't have a need for extra synchronization.
The combination of X=13 and MPI_F_SYNC_REG(X) before MPI_Barrier should guarantee that all bytes of X are stored in memory. The same should be valid in C, because the C compiler has no chance to see whether MPI_Barrier will access the bytes of X or not. And if it is guaranteed to be in the unified memory, then the other process (B) should be able to correctly read the data after the return from its barrier.
What is wrong with my thinking? Which detail do I miss?
According to my reading of the spec, MPI_F_SYNC_REG only prevents the language/compiler from moving the store, but does not say anything about processor ordering. So the WIN_SYNC in my last e-mail will add the processor memory barrier, which will give you all the semantics you need.
Shared memory programming is a disaster in most languages today, so we decided to pass that disaster on to the user. We really can't help, without adding lots of overhead (ie, using put/get/rma synchronization). So if a user already knows how to do shared memory programming, this will feel natural. If they don't, it's going to hurt badly :/.
Brian
-- Brian W. Barrett Scalable System Software Group Sandia National Laboratories
_______________________________________________ mpiwg-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
-- Dr. Rolf Rabenseifner . . . . . . . . . .. email [email protected] High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)