Pavan, please can you look at the more detailed question at end of question 1. Thank you for your answers on questions 2 and 3. Rolf ----- Original Message -----

From: "Pavan Balaji" <[email protected]> To: "MPI WG Remote Memory Access working group" <[email protected]> Cc: "Bill Long" <[email protected]> Sent: Tuesday, August 5, 2014 9:53:06 AM Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality

On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <[email protected]> wrote:

...

Dear expert on ccNUMA,

Huh?

...

1. Question (sequential consistency on one location): ------------

Do I understand correctly that in the following patter on a shared Memory or a ccNUM shared memory

rank 0 rank 1 print x x=val_1 print x x=val_2 print x print x

the print statements can print only in the following sequence - some times the previous value - some times val_1 - and after some time val_2 and it then stays to print val_2

and that it can never be that a sequence with val_2 before val_1 can be produced, i.e., old_val val_2 val_1 val_2 is impossible.

When there are dependencies between statements, the compiler and architecture don’t reorder them.

...

Also other values are impossible, e.g., some bit or byte-mix from val_1 and val_2.

That is possible, unless you use some form of atomicity.

In the case of automatic atomicity according to answer of 2nd question, together with the assumption that the compiler did not reorder (i.e. the stores are the store instructions and not source code lines), is it then guaranteed that the other process cannot see a different sequence, i.e., that it cannot see val_2 for a short time before it sees val_1, i.e., it is impossible that the other process would see

...

old_val val_2 val_1 val_2 isn't it?

...

2. Question: ----------- What is the largest size that the memory operations are atomic, i.e., that we do not see a bit or byte-mix from val_1 and val_2? Is it 1, 4, 8, 16 bytes or can it be a total struct that fits into a cacheline?

It depends on the architecture, but all architectures I’m aware of do give store atomicity of loads and stores till sizeof(void*), assuming many things including the variables are volatile, aligned correctly in memory, etc. Of course, this is not portable. To be portable, you should use atomic operations, e.g., with OpenPA.

...

3. Question (about two updates): -----------

rank 0 rank 1 x=x_ld y=yold ---- necessary synchronizations ----- print x (which shows xold) print y (which shows yold) ---- necessary synchronizations ----- x=xnew y=ynew print x print y after some time print x print y

Possible results are - xold,yold xold,yold xnew,ynew - xold,yold xnew,yold xnew,ynew - xold,yold xold,ynew xnew,ynew i.e., the y=ynew can arrive at another process faster than the x=xnew, although the storing process issues the stores in the sequence x=xnew, y=ynew. - xold,yold xnew,ynew xnew,ynew

The assignments should represent the store instructions, and not the source code (because the compiler may modify sequence of instructions compared to the source code)

Do I understand correctly, that the sequence of two store instructions two two different locations in one process may be visible at another process in a different sequence?

Correct. The compiler/hardware can reorder operations that they perceive as unrelated.

-- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

_______________________________________________ mpiwg-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

-- Dr. Rolf Rabenseifner . . . . . . . . . .. email [email protected] High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)