August 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by William Gropp 05 Aug '14

05 Aug '14

Question 1 is hard to answer. First, as it came out in my output, the first and last “print x” straddle the two ranks, so I’m not sure what that means. But the following is valid for the compiler: On rank 0. Since x = val_2 follows x = val_1 with no intervening use of x, a good optimizing compiler may eliminate the store to x with val_1 as unnecessary. On rank 1, Since x is never assigned in this block, a compiler is permitted to make a copy in a register (which would have the value x had before this block was entered) and than print that value each time print is called (e.g., it could build the call stack in memory, and just pass that unchanged call stack to print for each call). And don’t forget - there’s a difference between when the print completes on a process/thread and when the output appears - the system could flush all print output from rank 1 before any print output from rank 0 appears. If x is volatile, then the compiler isn’t free to do the second (save x and never load it again). Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Aug 5, 2014, at 2:53 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Huh? > >> 1. Question (sequential consistency on one location): >> ------------ >> >> Do I understand correctly that in the following patter >> on a shared Memory or a ccNUM shared memory >> >> rank 0 rank 1 >> print x >> x=val_1 print x >> x=val_2 print x >> print x >> >> the print statements can print only in the following >> sequence >> - some times the previous value >> - some times val_1 >> - and after some time val_2 and it then stays to print val_2 >> >> and that it can never be that a sequence with val_2 before val_1 >> can be produced, i.e., >> old_val >> val_2 >> val_1 >> val_2 >> is impossible. >

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

Pavan, please can you look at the more detailed question at end of question 1. Thank you for your answers on questions 2 and 3. Rolf ----- Original Message ----- > From: "Pavan Balaji" <balaji(a)anl.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Bill Long" <longb(a)cray.com> > Sent: Tuesday, August 5, 2014 9:53:06 AM > Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality > > > On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > > Dear expert on ccNUMA, > > Huh? > > > 1. Question (sequential consistency on one location): > > ------------ > > > > Do I understand correctly that in the following patter > > on a shared Memory or a ccNUM shared memory > > > > rank 0 rank 1 > > print x > > x=val_1 print x > > x=val_2 print x > > print x > > > > the print statements can print only in the following > > sequence > > - some times the previous value > > - some times val_1 > > - and after some time val_2 and it then stays to print val_2 > > > > and that it can never be that a sequence with val_2 before val_1 > > can be produced, i.e., > > old_val > > val_2 > > val_1 > > val_2 > > is impossible. > > When there are dependencies between statements, the compiler and > architecture don’t reorder them. > > > Also other values are impossible, e.g., some bit or byte-mix > > from val_1 and val_2. > > That is possible, unless you use some form of atomicity. In the case of automatic atomicity according to answer of 2nd question, together with the assumption that the compiler did not reorder (i.e. the stores are the store instructions and not source code lines), is it then guaranteed that the other process cannot see a different sequence, i.e., that it cannot see val_2 for a short time before it sees val_1, i.e., it is impossible that the other process would see > > old_val > > val_2 > > val_1 > > val_2 isn't it? > > 2. Question: > > ----------- > > What is the largest size that the memory operations are atomic, > > i.e., that we do not see a bit or byte-mix from val_1 and val_2? > > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits > > into a cacheline? > > It depends on the architecture, but all architectures I’m aware of do > give store atomicity of loads and stores till sizeof(void*), > assuming many things including the variables are volatile, aligned > correctly in memory, etc. Of course, this is not portable. To be > portable, you should use atomic operations, e.g., with OpenPA. > > > 3. Question (about two updates): > > ----------- > > > > rank 0 rank 1 > > x=x_ld > > y=yold > > ---- necessary synchronizations ----- > > print x (which shows xold) > > print y (which shows yold) > > ---- necessary synchronizations ----- > > x=xnew > > y=ynew > > print x > > print y > > after some time > > print x > > print y > > > > Possible results are > > - xold,yold xold,yold xnew,ynew > > - xold,yold xnew,yold xnew,ynew > > - xold,yold xold,ynew xnew,ynew > > i.e., the y=ynew can arrive at another process > > faster than the x=xnew, although the storing > > process issues the stores in the sequence > > x=xnew, y=ynew. > > - xold,yold xnew,ynew xnew,ynew > > > > The assignments should represent the store instructions, > > and not the source code (because the compiler may modify > > sequence of instructions compared to the source code) > > > > Do I understand correctly, that the sequence of two > > store instructions two two different locations in one process > > may be visible at another process in a different sequence? > > Correct. The compiler/hardware can reorder operations that they > perceive as unrelated. > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Balaji, Pavan 05 Aug '14

05 Aug '14

On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear expert on ccNUMA, Huh? > 1. Question (sequential consistency on one location): > ------------ > > Do I understand correctly that in the following patter > on a shared Memory or a ccNUM shared memory > > rank 0 rank 1 > print x > x=val_1 print x > x=val_2 print x > print x > > the print statements can print only in the following > sequence > - some times the previous value > - some times val_1 > - and after some time val_2 and it then stays to print val_2 > > and that it can never be that a sequence with val_2 before val_1 > can be produced, i.e., > old_val > val_2 > val_1 > val_2 > is impossible. When there are dependencies between statements, the compiler and architecture don’t reorder them. > Also other values are impossible, e.g., some bit or byte-mix > from val_1 and val_2. That is possible, unless you use some form of atomicity. > 2. Question: > ----------- > What is the largest size that the memory operations are atomic, > i.e., that we do not see a bit or byte-mix from val_1 and val_2? > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits > into a cacheline? It depends on the architecture, but all architectures I’m aware of do give store atomicity of loads and stores till sizeof(void*), assuming many things including the variables are volatile, aligned correctly in memory, etc. Of course, this is not portable. To be portable, you should use atomic operations, e.g., with OpenPA. > 3. Question (about two updates): > ----------- > > rank 0 rank 1 > x=x_ld > y=yold > ---- necessary synchronizations ----- > print x (which shows xold) > print y (which shows yold) > ---- necessary synchronizations ----- > x=xnew > y=ynew > print x > print y > after some time > print x > print y > > Possible results are > - xold,yold xold,yold xnew,ynew > - xold,yold xnew,yold xnew,ynew > - xold,yold xold,ynew xnew,ynew > i.e., the y=ynew can arrive at another process > faster than the x=xnew, although the storing > process issues the stores in the sequence > x=xnew, y=ynew. > - xold,yold xnew,ynew xnew,ynew > > The assignments should represent the store instructions, > and not the source code (because the compiler may modify > sequence of instructions compared to the source code) > > Do I understand correctly, that the sequence of two > store instructions two two different locations in one process > may be visible at another process in a different sequence? Correct. The compiler/hardware can reorder operations that they perceive as unrelated. -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Short question on the ccNUMA memory reality
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

Dear expert on ccNUMA, three questions, which hopefully may be trivial: 1. Question (sequential consistency on one location): ------------ Do I understand correctly that in the following patter on a shared Memory or a ccNUM shared memory rank 0 rank 1 print x x=val_1 print x x=val_2 print x print x the print statements can print only in the following sequence - some times the previous value - some times val_1 - and after some time val_2 and it then stays to print val_2 and that it can never be that a sequence with val_2 before val_1 can be produced, i.e., old_val val_2 val_1 val_2 is impossible. Also other values are impossible, e.g., some bit or byte-mix from val_1 and val_2. 2. Question: ----------- What is the largest size that the memory operations are atomic, i.e., that we do not see a bit or byte-mix from val_1 and val_2? Is it 1, 4, 8, 16 bytes or can it be a total struct that fits into a cacheline? 3. Question (about two updates): ----------- rank 0 rank 1 x=x_ld y=yold ---- necessary synchronizations ----- print x (which shows xold) print y (which shows yold) ---- necessary synchronizations ----- x=xnew y=ynew print x print y after some time print x print y Possible results are - xold,yold xold,yold xnew,ynew - xold,yold xnew,yold xnew,ynew - xold,yold xold,ynew xnew,ynew i.e., the y=ynew can arrive at another process faster than the x=xnew, although the storing process issues the stores in the sequence x=xnew, y=ynew. - xold,yold xnew,ynew xnew,ynew The assignments should represent the store instructions, and not the source code (because the compiler may modify sequence of instructions compared to the source code) Do I understand correctly, that the sequence of two store instructions two two different locations in one process may be visible at another process in a different sequence? I ask all these questions to understand which memory model can be defined for MPI shared memory windows. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [MPI Forum] #437: MPI 3.0 Errata: MPI Active Target synchronization requirements for shared memory, 2 alternatives
by Rolf Rabenseifner 04 Aug '14

04 Aug '14

Dear Bill, will you be at the Japan Forum meeting? in our telecon - provided I understood correctly - you told that you'll try to make a first proposal to solve the shared memory problems. Please take this email only as a possible input for your proposal, i.e., it is not thought as a discussion base. For me, it is important to finish this discussion more than two weeks before the Japan meeting, because it must be part of the MPI-3.0 errata and MPI-3.1. The two-week deadline is already Sep.1, 2014, i.e., 4 weeks from now! ----- Important sections that may need corrections: Sect.11.2.3 p409:16-17 mentions the first time "The locally allocated memory can be the target of load/store accesses by remote processes;" Here we may mention: The rules for RMA operation do not apply to these these remote load/store accesses; additional rules apply, see, e.g., Section 11.5.4A [the new one below] and 11.5.5 [on assertions] on pages ... and ... . p410:15-21: May need corrections p436:37-48: Especially the first sentence - #436 How is this section related to shared Memory. If there are additional rules for shared memory that do not apply to all unified memory Windows, then an additional section on "shared & unified" memory need to be added. I recommend to define it in the way "a load/store access in one process ... a load/store access in another process" and not in the way of "local" and "remote load/store". For me, after the MPI_Win_allocate_shared, there is not really memory portions that are local to the processes. The sentence p436:45-46 "The order in which data is written is not specified unless further synchronization is used." If a process issues several load and stores to the same location, they are executed in the given sequence without further need of synchronization rank 0 rank 1 x=1 print x x=2 print x If two processes access the same location, may it possible that the three prints may be other than 1-1 1-2 2-2? Do we want to keep this undefined? I would propose to add a rationale like --- Rationale. If two processes on a ccNUMA node access the same memory location with an intermediate process-to-process synchronization then the outcome is still undefined because the first memory operation need not to be finished before the second memory operation starts. A well-defined execution may require the following sequence: 1) the memory access on the first process; 2) a local memory fence to guarantee that the memory operation is finished; 3) the process-to-process synchronization (e.g. send/recv) to inform the second process; 4) a local memory fence on the second process to guarantee that subsequent memory operations are issued to the memory 5) the memory operation on the second process. To guarantee the outcome of memory accesses in combination with the synchronization as described in Section 11.5.4A on page ... (NEW SECTION, SEE BELOW), the synchronization operation will issue such local memory fences if needed. End of rationale. --- This rationale makes it understandable, what we discussed in such many emails about the need of MPI_WIN_SYNC. The major part are the new synchronization rules: I propose to put them together in one new section after 11.5.4 11.5.4A Shared Memory Synchronization In the case of an MPI shared memory window additional rules apply for synchronizing load and store accesses from several processes to the same location. A location must be at least a byte. In the following patterns, locations in a shared memory window are noted with variable names A, B, C, loads from such windows are noted with load(...) and stores are noted by assignments to these variables. Patterns with active target communication and MPI_Win_sync: process P0 process P1 A=val_1 Sync-to-P1 --> Sync-from-P0 load(A) load(B) Sync-to-P1 --> Sync-from-P0 B=val_2 C=val_3 Sync-to-P1 --> Sync-from-P0 C=val_4 load (C) with Sync-to-P1 --> Sync-from-P0 can be 1. MPI_Win_fence --> MPI_Win_fence 1) 2. MPI_Win_post --> MPI_Win_start 2) 3. MPI_Win_complete--> MPI_Win_wait 3) 4. MPI_Win_sync any process sync to P1 --> any process sync from P0 4) MPI_Win_sync Footnotes: 1) MPI_Win_fence synchronizes in both directions and between every process in the process group of the window. 2) The arrow means that P1 is in the origin group passed to MPI_Win_post in P0, and that P0 is in the target group passed to MPI_Win_start. Additional calls to MPI_Win_complete (in P1 after MPI_Win_start) and MPI_Win_wait (in P0 after MPI_Win_post) are need. The location of these calls do not influence the guaranteed outcome rules. 3) The arrow means that P1 is in the target group passed to MPI_Win_start that corresponds to MPI_Win_complete in P0, and P0 is in the origin group passed to MPI_Win_post that corresponds to MPI_Win_wait in P1. Additional calls to MPI_Win_start (in P0 before MPI_Win_complete) and MPI_Win_post (in P1 before MPI_Win_wait) are need. The location of these calls do not influence the guaranteed outcome rules. 4) The synchronization may be done with methods from MPI (e.g. send-->recv) or with other methods. The requirements for using MPI_Win_sync (e.g. within a passive target epoch which may be provided with MPI_Win_lock_all) are not shown in thee pattern Patterns with lock/unlock synchronization: Within passive target communication, two locks L1 and L2 may be scheduled L1 before L2 or L2 before L1. In the following patterns, the arrow means that the lock in P0 was scheduled before the lock in P1. For the following load/store/sync patterns process P0 process P1 MPI_Win_lock exclusive A=val_1 MPI_Win_Unlock --> MPI_Lock shared or exclusive load(A) MPI_Win_lock MPI_Win_lock shared or exclusive load(B) MPI_Win_Unlock --> MPI_Lock exclusive B=val_2 MPI_Win_lock MPI_Win_lock exclusive C=val_3 MPI_Win_Unlock --> MPI_Lock exclusive C=val_4 load (C) MPI_Win_lock Note that each rank of a window is connected to a separate lock. In a shared window, these locks are not connected to specific memory portions of the shared window, i.e., each lock can be used to protect any portion of a shared memory window. In the patterns above, it is guaranteed - that the load(A) in P1 loads val_1 (this is the Write-read-rule), - that the load(B) in P0 is not affected by the store of val_2 in P1 (read-write-rule), and - that the load(C) in P1 loads val_4 (write-write-rule). ---- I do not understand which rules are guaranteed for - MPI_Win_flush (and MPI_Win_flush_all) - MPI_Win_flush_local (and MPI_Win_flush_local_all) I hope that somebody understands these calls and can produce the related patterns. ---- About Sect. 11.5.5 Assertions - MPI_MODE_NOSTORE: I expect that the proposal in #429 is not helpful: it enlarges the hint about "not updated by stores since last synchronization" from local stores to all stores by the whole process group. The reason for this hint is to prevent "cache synchronization". This cache synchronization is local and therefore the remote stores do not count. The new proposal is simple: p452:1 and p452.9: "stores" --> "local stores" Additionally, I would add the following sentence: "In the case of a shared memory window, such local stores can be issued to any portion of the shared memory window." Reason: Nobody should think that "local" means "only to the window portion that was defined in the local MPI_WIN_ALLOCATE_SHARED". Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0