- mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] [MPI Forum] #437: MPI 3.0 Errata: MPI Active Target synchronization requirements for shared memory, 2 alternatives
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

(Small update below in case 4. It is split to 4AC and 4B.) Dear Bill, will you be at the Japan Forum meeting? in our telecon - provided I understood correctly - you told that you'll try to make a first proposal to solve the shared memory problems. Please take this email only as a possible input for your proposal, i.e., it is not thought as a discussion base. For me, it is important to finish this discussion more than two weeks before the Japan meeting, because it must be part of the MPI-3.0 errata and MPI-3.1. The two-week deadline is already Sep.1, 2014, i.e., 4 weeks from now! ----- Important sections that may need corrections: Sect.11.2.3 p409:16-17 mentions the first time "The locally allocated memory can be the target of load/store accesses by remote processes;" Here we may mention: The rules for RMA operation do not apply to these these remote load/store accesses; additional rules apply, see, e.g., Section 11.5.4A [the new one below] and 11.5.5 [on assertions] on pages ... and ... . p410:15-21: May need corrections p436:37-48: Especially the first sentence - #436 How is this section related to shared Memory. If there are additional rules for shared memory that do not apply to all unified memory Windows, then an additional section on "shared & unified" memory need to be added. I recommend to define it in the way "a load/store access in one process ... a load/store access in another process" and not in the way of "local" and "remote load/store". For me, after the MPI_Win_allocate_shared, there is not really memory portions that are local to the processes. The sentence p436:45-46 "The order in which data is written is not specified unless further synchronization is used." If a process issues several load and stores to the same location, they are executed in the given sequence without further need of synchronization rank 0 rank 1 x=1 print x x=2 print x If two processes access the same location, may it possible that the three prints may be other than 1-1 1-2 2-2? Do we want to keep this undefined? I would propose to add a rationale like --- Rationale. If two processes on a ccNUMA node access the same memory location with an intermediate process-to-process synchronization then the outcome is still undefined because the first memory operation need not to be finished before the second memory operation starts. A well-defined execution may require the following sequence: 1) the memory access on the first process; 2) a local memory fence to guarantee that the memory operation is finished; 3) the process-to-process synchronization (e.g. send/recv) to inform the second process; 4) a local memory fence on the second process to guarantee that subsequent memory operations are issued to the memory 5) the memory operation on the second process. To guarantee the outcome of memory accesses in combination with the synchronization as described in Section 11.5.4A on page ... (NEW SECTION, SEE BELOW), the synchronization operation will issue such local memory fences if needed. End of rationale. --- This rationale makes it understandable, what we discussed in such many emails about the need of MPI_WIN_SYNC. The major part are the new synchronization rules: I propose to put them together in one new section after 11.5.4 11.5.4A Shared Memory Synchronization In the case of an MPI shared memory window additional rules apply for synchronizing load and store accesses from several processes to the same location. A location must be at least a byte. In the following patterns, locations in a shared memory window are noted with variable names A, B, C, loads from such windows are noted with load(...) and stores are noted by assignments to these variables. Patterns with active target communication and MPI_Win_sync: process P0 process P1 A=val_1 Sync-to-P1 --> Sync-from-P0 load(A) load(B) Sync-to-P1 --> Sync-from-P0 B=val_2 C=val_3 Sync-to-P1 --> Sync-from-P0 C=val_4 load (C) with Sync-to-P1 --> Sync-from-P0 can be 1. MPI_Win_fence --> MPI_Win_fence 1) 2. MPI_Win_post --> MPI_Win_start 2) 3. MPI_Win_complete--> MPI_Win_wait 3) 4AC. Only for the cases with variables A (i.e., write-read) and C (i.e,, write-write): MPI_Win_sync any process sync from P0 to P1 --> any process sync from P0 to 4) MPI_Win_sync 4B. Only for the case with variable B (i.e., read-write rule) any process sync from P0 to P1 --> any process sync from P0 to 4) Footnotes: 1) MPI_Win_fence synchronizes in both directions and between every process in the process group of the window. 2) The arrow means that P1 is in the origin group passed to MPI_Win_post in P0, and that P0 is in the target group passed to MPI_Win_start. Additional calls to MPI_Win_complete (in P1 after MPI_Win_start) and MPI_Win_wait (in P0 after MPI_Win_post) are need. The location of these calls do not influence the guaranteed outcome rules. 3) The arrow means that P1 is in the target group passed to MPI_Win_start that corresponds to MPI_Win_complete in P0, and P0 is in the origin group passed to MPI_Win_post that corresponds to MPI_Win_wait in P1. Additional calls to MPI_Win_start (in P0 before MPI_Win_complete) and MPI_Win_post (in P1 before MPI_Win_wait) are need. The location of these calls do not influence the guaranteed outcome rules. 4) The synchronization may be done with methods from MPI (e.g. send-->recv) or with other methods. The requirements for using MPI_Win_sync (e.g. within a passive target epoch which may be provided with MPI_Win_lock_all) are not shown in thee pattern Patterns with lock/unlock synchronization: Within passive target communication, two locks L1 and L2 may be scheduled L1 before L2 or L2 before L1. In the following patterns, the arrow means that the lock in P0 was scheduled before the lock in P1. For the following load/store/sync patterns process P0 process P1 MPI_Win_lock exclusive A=val_1 MPI_Win_Unlock --> MPI_Lock shared or exclusive load(A) MPI_Win_lock MPI_Win_lock shared or exclusive load(B) MPI_Win_Unlock --> MPI_Lock exclusive B=val_2 MPI_Win_lock MPI_Win_lock exclusive C=val_3 MPI_Win_Unlock --> MPI_Lock exclusive C=val_4 load (C) MPI_Win_lock Note that each rank of a window is connected to a separate lock. In a shared window, these locks are not connected to specific memory portions of the shared window, i.e., each lock can be used to protect any portion of a shared memory window. In the patterns above, it is guaranteed - that the load(A) in P1 loads val_1 (this is the Write-read-rule), - that the load(B) in P0 is not affected by the store of val_2 in P1 (read-write-rule), and - that the load(C) in P1 loads val_4 (write-write-rule). ---- I do not understand which rules are guaranteed for - MPI_Win_flush (and MPI_Win_flush_all) - MPI_Win_flush_local (and MPI_Win_flush_local_all) I hope that somebody understands these calls and can produce the related patterns. ---- About Sect. 11.5.5 Assertions - MPI_MODE_NOSTORE: I expect that the proposal in #429 is not helpful: it enlarges the hint about "not updated by stores since last synchronization" from local stores to all stores by the whole process group. The reason for this hint is to prevent "cache synchronization". This cache synchronization is local and therefore the remote stores do not count. The new proposal is simple: p452:1 and p452.9: "stores" --> "local stores" Additionally, I would add the following sentence: "In the case of a shared memory window, such local stores can be issued to any portion of the shared memory window." Reason: Nobody should think that "local" means "only to the window portion that was defined in the local MPI_WIN_ALLOCATE_SHARED". Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Dave Goodell (dgoodell) 05 Aug '14

05 Aug '14

On Aug 5, 2014, at 2:53 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > >> 1. Question (sequential consistency on one location): >> ------------ >> >> Do I understand correctly that in the following patter >> on a shared Memory or a ccNUM shared memory >> >> rank 0 rank 1 >> print x >> x=val_1 print x >> x=val_2 print x >> print x >> >> the print statements can print only in the following >> sequence >> - some times the previous value >> - some times val_1 >> - and after some time val_2 and it then stays to print val_2 >> >> and that it can never be that a sequence with val_2 before val_1 >> can be produced, i.e., >> old_val >> val_2 >> val_1 >> val_2 >> is impossible. > > When there are dependencies between statements, the compiler and architecture don’t reorder them. Rolf, for a bit of clarification on this, I use Paul McKenney's theoretical "Ordering-Hostile Architecture" [1] as my guideline for what an arbitrary architecture might choose to do. ***This is separate from any language/compiler (non-)guarantees.*** [1] http://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook-e1.pdf """ C.6.1 Ordering-Hostile Architecture A number of ordering-hostile computer systems have been produced over the decades, but the nature of the hostility has always been extremely subtle, and understanding it has required detailed knowledge of the specific hardware. Rather than picking on a specific hardware vendor, and as a presumably attractive alternative to dragging the reader through detailed technical specifications, let us instead de- sign a mythical but maximally memory-ordering-hostile computer architecture.4 This hardware must obey the following ordering con- straints [McK05a, McK05b]: 1. Each CPU will always perceive its own memory accesses as occurring in program order. 2. CPUs will reorder a given operation with a store only if the two operations are referencing different locations. 3. All of a given CPU’s loads preceding a read memory barrier (smp_rmb()) will be perceived by all CPUs to precede any loads following that read memory barrier. 4. All of a given CPU’s stores preceding a write mem- ory barrier (smp_wmb()) will be perceived by all CPUs to precede any stores following that write memory barrier. 5. All of a given CPU’s accesses (loads and stores) preceding a full memory barrier (smp_mb()) will be perceived by all CPUs to precede any accesses following that memory barrier. """ Point 2 is particularly relevant here. >> Also other values are impossible, e.g., some bit or byte-mix >> from val_1 and val_2. > > That is possible, unless you use some form of atomicity. Right, though such "weird" patterns are most likely to occur due to some compiler behavior, not at the architectural level, assuming that "x" is word-sized or smaller and naturally aligned. -Dave

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

Bronis and all, yes, after all what I now know, I expect that you are right. My proposal for Bill G. was based on guarantees from rank to rank only with synchronizations in between. Thanks for all the answers. Rolf ----- Original Message ----- > From: "Bronis R. de Supinski" <bronis(a)llnl.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Bill Long" <longb(a)cray.com> > Sent: Tuesday, August 5, 2014 5:00:15 PM > Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality > > > Rolf: > > Re: > > thank you for your answer. > > I was not precise enough with my question. > > My notations should represent store and load instructions, > > i.e., assembler level, i.e., what's going on hardware level. > > The "level" is not really relevant. As Pavan noted, unless you use > atomic instructions, you get no guarantees. With ordinary > loads and stores, your snippets all contain race conditions. > The exact result will depend on the system but as a rule, > you can assume that a program with races provides NO guarantees. > > Sure, you can eliminate the values coming from registers but > they can come from the cache and you are not provide any > general atomicity guarantees. Most hardware will provide > bit-level atomicity on ordinary loads and stores but if > you straddle a cache line, you lose those guarantees. > > Overall, the issues that arise are more complex than you > want to deal with. That's why we have compilers for threaded > programs. Simply put, shared memory programming without compiler > assistance is not something ordinary programmers should do. > > Bronis > > > > x should be part of an MPI shared memory allocated with > > MPI_Win_allocate_shared, i.e., the assembler on rank 0 > > does two stores and the assembler on rank 1 does four loads. > > > > The code was > >> rank 0 rank 1 > >> _______ print x > >> x=val_1 print x > >> x=val_2 print x > >> _______ print x > > > > When I understand Bill Long correctly, then the 4 load(x) > > end up in > >> old_val > >> val_2 (because it was going through a very slow network path) > >> val_1 > >> val_2 > > wheras three enterleaved load instructions on rank 0 > > (i.e., in the same execution stream (thread) as the two stores) > > will always see > > old_val > > val_1 > > val_2 > > > > This is independent of compilers, because I only want to > > look at assembbler level. > > > > Bill Long, are you sure? I would expect that all loads go through > > the > > 1st Level Cache and as soon as it sees val_2 it should not be > > possible to see with a later issued instruction val_1. > > > > Best regards > > Rolf > > > > > > ----- Original Message ----- > >> From: "Bill Long" <longb(a)cray.com> > >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >> Cc: "MPI WG Remote Memory Access working group" > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> Sent: Tuesday, August 5, 2014 3:12:58 PM > >> Subject: Re: Short question on the ccNUMA memory reality > >> > >> Hi Rolf, > >> > >> > >> I assume you are expecting answers from people like Pavan and Bill > >> G > >> for the MPI RMA perspective, and for the Fortran rules from me. > >> (For > >> Fortran, map Rank 0 -> image 1 and Rank 1 -> Image 2). > >> > >> On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner > >> <rabenseifner(a)hlrs.de> > >> wrote: > >> > >> > Dear expert on ccNUMA, > >> > > >> > three questions, which hopefully may be trivial: > >> > > >> > 1. Question (sequential consistency on one location): > >> > ------------ > >> > > >> > Do I understand correctly that in the following patter > >> > on a shared Memory or a ccNUM shared memory > >> > >> I assume you mean what I think of as distributed memory here. > >> Otherwise, these are questions about OpenMP, and the rank 0 / > >> rank > >> 1 separation does not make sense. > >> > >> > > >> > rank 0 rank 1 > >> > print x > >> > x=val_1 print x > >> > x=val_2 print x > >> > print x > >> > >> This depends on whether x is declared in a way that makes it > >> accessible from a remote rank. If not, then the code is illegal. > >> So > >> I’ll assume it is accessible. > >> > >> Since there is no synchronization between rank 0 and rank 1, > >> compiler > >> elimination of the x = val_1 assignment is allowed (and likely > >> expected). > >> > >> Even if the assignment is not eliminated, the print x on rank 1 > >> involves a “get” of x from rank 0. Assuming it is declared > >> volatile, to eliminate the “snap to local temp” optimization on > >> rank > >> 1, it is still possible to get the values out of order. For > >> example, the print #2 could get routed through the network via > >> some > >> extra long path, while print #3 could get routed on the fastest > >> path, resulting in the two “get” operations appearing at rank 0 in > >> an unpredictable order. > >> > >> [Note that by the Fortran rules, the code is not legal anyway. If > >> you define a variable on one rank and reference it (the print > >> counts > >> as a reference) on another rank the execution segments containing > >> the definition and reference have to be ordered unless the > >> definition and reference are both explicitly atomic. Segment > >> ordering is done with synchronization statements.] > >> > >> > >> > > >> > the print statements can print only in the following > >> > sequence > >> > - some times the previous value > >> > - some times val_1 > >> > - and after some time val_2 and it then stays to print val_2 > >> > > >> > and that it can never be that a sequence with val_2 before val_1 > >> > can be produced, i.e., > >> > old_val > >> > val_2 > >> > val_1 > >> > val_2 > >> > is impossible. > >> > > >> > Also other values are impossible, e.g., some bit or byte-mix > >> > from val_1 and val_2. > >> > >> > >> On most hardware, assuming the values are “normal” size, such as > >> 64-bit, you will not get mixed bits. However, if you want to > >> ensure > >> that, use the atomic get and put intrinsics. If the accesses had > >> been properly ordered by synchronizations, then you do not need > >> the > >> atomics to ensure whole values. > >> > >> > >> > > >> > 2. Question: > >> > ----------- > >> > What is the largest size that the memory operations are atomic, > >> > i.e., that we do not see a bit or byte-mix from val_1 and val_2? > >> > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits > >> > into a cacheline? > >> > >> > >> Since the types in Fortran are parameterized, the processor > >> supplies > >> a KIND value for the integer and logical types for which atomic > >> operations are supported. Typically kind corresponds to 64 bits, > >> but could be 32 on some architectures. > >> > >> > >> > > >> > 3. Question (about two updates): > >> > ----------- > >> > > >> > rank 0 rank 1 > >> > x=x_ld > >> > y=yold > >> > ---- necessary synchronizations ----- > >> > print x (which shows xold) > >> > print y (which shows yold) > >> > ---- necessary synchronizations ----- > >> > x=xnew > >> > y=ynew > >> > print x > >> > print y > >> > after some time > >> > print x > >> > print y > >> > > >> > Possible results are > >> > - xold,yold xold,yold xnew,ynew > >> > - xold,yold xnew,yold xnew,ynew > >> > - xold,yold xold,ynew xnew,ynew > >> > i.e., the y=ynew can arrive at another process > >> > faster than the x=xnew, although the storing > >> > process issues the stores in the sequence > >> > x=xnew, y=ynew. > >> > - xold,yold xnew,ynew xnew,ynew > >> > > >> > The assignments should represent the store instructions, > >> > and not the source code (because the compiler may modify > >> > sequence of instructions compared to the source code) > >> > > >> > Do I understand correctly, that the sequence of two > >> > store instructions two two different locations in one process > >> > may be visible at another process in a different sequence? > >> > >> Correct. As before, the “get” operations implied in the print > >> statements could arrive out of order. Also, while unlikely, the > >> compiler could decide to reverse the order of the assignments > >> since > >> they are independent. (Assuming x and y cannot be aliased by, for > >> example, being pointers.) > >> > >> Cheers, > >> Bill > >> > >> > > >> > I ask all these questions to understand which memory model > >> > can be defined for MPI shared memory windows. > >> > > >> > Best regards > >> > Rolf > >> > > >> > -- > >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email > >> > rabenseifner(a)hlrs.de > >> > High Performance Computing Center (HLRS) . phone > >> > ++49(0)711/685-65530 > >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >> > 685-65832 > >> > Head of Dpmt Parallel Computing . . . > >> > www.hlrs.de/people/rabenseifner > >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >> > 1.307) > >> > >> Bill Long > >> longb(a)cray.com > >> Fortran Technical Suport & > >> voice: > >> 651-605-9024 > >> Bioinformatics Software Development fax: > >> 651-605-9142 > >> Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN > >> 55101 > >> > >> > > > > > > ----- Original Message ----- > >> From: "William Gropp" <wgropp(a)illinois.edu> > >> To: "MPI WG Remote Memory Access working group" > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> Cc: "Bill Long" <longb(a)cray.com> > >> Sent: Tuesday, August 5, 2014 2:38:07 PM > >> Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory > >> reality > >> > >> > >> Question 1 is hard to answer. First, as it came out in my output, > >> the first and last “print x” straddle the two ranks, so I’m not > >> sure > >> what that means. But the following is valid for the compiler: > >> > >> > >> On rank 0. Since x = val_2 follows x = val_1 with no intervening > >> use > >> of x, a good optimizing compiler may eliminate the store to x with > >> val_1 as unnecessary. > >> > >> > >> On rank 1, Since x is never assigned in this block, a compiler is > >> permitted to make a copy in a register (which would have the value > >> x > >> had before this block was entered) and than print that value each > >> time print is called (e.g., it could build the call stack in > >> memory, and just pass that unchanged call stack to print for each > >> call). > >> > >> > >> And don’t forget - there’s a difference between when the print > >> completes on a process/thread and when the output appears - the > >> system could flush all print output from rank 1 before any print > >> output from rank 0 appears. > >> > >> > >> If x is volatile, then the compiler isn’t free to do the second > >> (save > >> x and never load it again). > >> > >> > >> Bill > >> > >> > >> > >> > >> > >> > >> William Gropp > >> Director, Parallel Computing Institute Thomas M. Siebel Chair in > >> Computer Science > >> > >> > >> University of Illinois Urbana-Champaign > >> > >> > >> > >> > >> > >> > >> > >> On Aug 5, 2014, at 2:53 AM, Balaji, Pavan < balaji(a)anl.gov > > >> wrote: > >> > >> > >> Huh? > >> > >> > >> > >> 1. Question (sequential consistency on one location): > >> ------------ > >> > >> Do I understand correctly that in the following patter > >> on a shared Memory or a ccNUM shared memory > >> > >> rank 0 rank 1 > >> print x > >> x=val_1 print x > >> x=val_2 print x > >> print x > >> > >> the print statements can print only in the following > >> sequence > >> - some times the previous value > >> - some times val_1 > >> - and after some time val_2 and it then stays to print val_2 > >> > >> and that it can never be that a sequence with val_2 before val_1 > >> can be produced, i.e., > >> old_val > >> val_2 > >> val_1 > >> val_2 > >> is impossible. > >> > >> > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Bronis R. de Supinski 05 Aug '14

05 Aug '14

Rolf: Re: > thank you for your answer. > I was not precise enough with my question. > My notations should represent store and load instructions, > i.e., assembler level, i.e., what's going on hardware level. The "level" is not really relevant. As Pavan noted, unless you use atomic instructions, you get no guarantees. With ordinary loads and stores, your snippets all contain race conditions. The exact result will depend on the system but as a rule, you can assume that a program with races provides NO guarantees. Sure, you can eliminate the values coming from registers but they can come from the cache and you are not provide any general atomicity guarantees. Most hardware will provide bit-level atomicity on ordinary loads and stores but if you straddle a cache line, you lose those guarantees. Overall, the issues that arise are more complex than you want to deal with. That's why we have compilers for threaded programs. Simply put, shared memory programming without compiler assistance is not something ordinary programmers should do. Bronis > x should be part of an MPI shared memory allocated with > MPI_Win_allocate_shared, i.e., the assembler on rank 0 > does two stores and the assembler on rank 1 does four loads. > > The code was >> rank 0 rank 1 >> _______ print x >> x=val_1 print x >> x=val_2 print x >> _______ print x > > When I understand Bill Long correctly, then the 4 load(x) > end up in >> old_val >> val_2 (because it was going through a very slow network path) >> val_1 >> val_2 > wheras three enterleaved load instructions on rank 0 > (i.e., in the same execution stream (thread) as the two stores) > will always see > old_val > val_1 > val_2 > > This is independent of compilers, because I only want to > look at assembbler level. > > Bill Long, are you sure? I would expect that all loads go through the > 1st Level Cache and as soon as it sees val_2 it should not be > possible to see with a later issued instruction val_1. > > Best regards > Rolf > > > ----- Original Message ----- >> From: "Bill Long" <longb(a)cray.com> >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >> Cc: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Tuesday, August 5, 2014 3:12:58 PM >> Subject: Re: Short question on the ccNUMA memory reality >> >> Hi Rolf, >> >> >> I assume you are expecting answers from people like Pavan and Bill G >> for the MPI RMA perspective, and for the Fortran rules from me. (For >> Fortran, map Rank 0 -> image 1 and Rank 1 -> Image 2). >> >> On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> >> wrote: >> >> > Dear expert on ccNUMA, >> > >> > three questions, which hopefully may be trivial: >> > >> > 1. Question (sequential consistency on one location): >> > ------------ >> > >> > Do I understand correctly that in the following patter >> > on a shared Memory or a ccNUM shared memory >> >> I assume you mean what I think of as distributed memory here. >> Otherwise, these are questions about OpenMP, and the rank 0 / rank >> 1 separation does not make sense. >> >> > >> > rank 0 rank 1 >> > print x >> > x=val_1 print x >> > x=val_2 print x >> > print x >> >> This depends on whether x is declared in a way that makes it >> accessible from a remote rank. If not, then the code is illegal. So >> I’ll assume it is accessible. >> >> Since there is no synchronization between rank 0 and rank 1, compiler >> elimination of the x = val_1 assignment is allowed (and likely >> expected). >> >> Even if the assignment is not eliminated, the print x on rank 1 >> involves a “get” of x from rank 0. Assuming it is declared >> volatile, to eliminate the “snap to local temp” optimization on rank >> 1, it is still possible to get the values out of order. For >> example, the print #2 could get routed through the network via some >> extra long path, while print #3 could get routed on the fastest >> path, resulting in the two “get” operations appearing at rank 0 in >> an unpredictable order. >> >> [Note that by the Fortran rules, the code is not legal anyway. If >> you define a variable on one rank and reference it (the print counts >> as a reference) on another rank the execution segments containing >> the definition and reference have to be ordered unless the >> definition and reference are both explicitly atomic. Segment >> ordering is done with synchronization statements.] >> >> >> > >> > the print statements can print only in the following >> > sequence >> > - some times the previous value >> > - some times val_1 >> > - and after some time val_2 and it then stays to print val_2 >> > >> > and that it can never be that a sequence with val_2 before val_1 >> > can be produced, i.e., >> > old_val >> > val_2 >> > val_1 >> > val_2 >> > is impossible. >> > >> > Also other values are impossible, e.g., some bit or byte-mix >> > from val_1 and val_2. >> >> >> On most hardware, assuming the values are “normal” size, such as >> 64-bit, you will not get mixed bits. However, if you want to ensure >> that, use the atomic get and put intrinsics. If the accesses had >> been properly ordered by synchronizations, then you do not need the >> atomics to ensure whole values. >> >> >> > >> > 2. Question: >> > ----------- >> > What is the largest size that the memory operations are atomic, >> > i.e., that we do not see a bit or byte-mix from val_1 and val_2? >> > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits >> > into a cacheline? >> >> >> Since the types in Fortran are parameterized, the processor supplies >> a KIND value for the integer and logical types for which atomic >> operations are supported. Typically kind corresponds to 64 bits, >> but could be 32 on some architectures. >> >> >> > >> > 3. Question (about two updates): >> > ----------- >> > >> > rank 0 rank 1 >> > x=x_ld >> > y=yold >> > ---- necessary synchronizations ----- >> > print x (which shows xold) >> > print y (which shows yold) >> > ---- necessary synchronizations ----- >> > x=xnew >> > y=ynew >> > print x >> > print y >> > after some time >> > print x >> > print y >> > >> > Possible results are >> > - xold,yold xold,yold xnew,ynew >> > - xold,yold xnew,yold xnew,ynew >> > - xold,yold xold,ynew xnew,ynew >> > i.e., the y=ynew can arrive at another process >> > faster than the x=xnew, although the storing >> > process issues the stores in the sequence >> > x=xnew, y=ynew. >> > - xold,yold xnew,ynew xnew,ynew >> > >> > The assignments should represent the store instructions, >> > and not the source code (because the compiler may modify >> > sequence of instructions compared to the source code) >> > >> > Do I understand correctly, that the sequence of two >> > store instructions two two different locations in one process >> > may be visible at another process in a different sequence? >> >> Correct. As before, the “get” operations implied in the print >> statements could arrive out of order. Also, while unlikely, the >> compiler could decide to reverse the order of the assignments since >> they are independent. (Assuming x and y cannot be aliased by, for >> example, being pointers.) >> >> Cheers, >> Bill >> >> > >> > I ask all these questions to understand which memory model >> > can be defined for MPI shared memory windows. >> > >> > Best regards >> > Rolf >> > >> > -- >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email >> > rabenseifner(a)hlrs.de >> > High Performance Computing Center (HLRS) . phone >> > ++49(0)711/685-65530 >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >> > 685-65832 >> > Head of Dpmt Parallel Computing . . . >> > www.hlrs.de/people/rabenseifner >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >> > 1.307) >> >> Bill Long >> longb(a)cray.com >> Fortran Technical Suport & voice: >> 651-605-9024 >> Bioinformatics Software Development fax: >> 651-605-9142 >> Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101 >> >> > > > ----- Original Message ----- >> From: "William Gropp" <wgropp(a)illinois.edu> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Cc: "Bill Long" <longb(a)cray.com> >> Sent: Tuesday, August 5, 2014 2:38:07 PM >> Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality >> >> >> Question 1 is hard to answer. First, as it came out in my output, >> the first and last “print x” straddle the two ranks, so I’m not sure >> what that means. But the following is valid for the compiler: >> >> >> On rank 0. Since x = val_2 follows x = val_1 with no intervening use >> of x, a good optimizing compiler may eliminate the store to x with >> val_1 as unnecessary. >> >> >> On rank 1, Since x is never assigned in this block, a compiler is >> permitted to make a copy in a register (which would have the value x >> had before this block was entered) and than print that value each >> time print is called (e.g., it could build the call stack in >> memory, and just pass that unchanged call stack to print for each >> call). >> >> >> And don’t forget - there’s a difference between when the print >> completes on a process/thread and when the output appears - the >> system could flush all print output from rank 1 before any print >> output from rank 0 appears. >> >> >> If x is volatile, then the compiler isn’t free to do the second (save >> x and never load it again). >> >> >> Bill >> >> >> >> >> >> >> William Gropp >> Director, Parallel Computing Institute Thomas M. Siebel Chair in >> Computer Science >> >> >> University of Illinois Urbana-Champaign >> >> >> >> >> >> >> >> On Aug 5, 2014, at 2:53 AM, Balaji, Pavan < balaji(a)anl.gov > wrote: >> >> >> Huh? >> >> >> >> 1. Question (sequential consistency on one location): >> ------------ >> >> Do I understand correctly that in the following patter >> on a shared Memory or a ccNUM shared memory >> >> rank 0 rank 1 >> print x >> x=val_1 print x >> x=val_2 print x >> print x >> >> the print statements can print only in the following >> sequence >> - some times the previous value >> - some times val_1 >> - and after some time val_2 and it then stays to print val_2 >> >> and that it can never be that a sequence with val_2 before val_1 >> can be produced, i.e., >> old_val >> val_2 >> val_1 >> val_2 >> is impossible. >> >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

Bill & Bill, thank you for your answer. I was not precise enough with my question. My notations should represent store and load instructions, i.e., assembler level, i.e., what's going on hardware level. x should be part of an MPI shared memory allocated with MPI_Win_allocate_shared, i.e., the assembler on rank 0 does two stores and the assembler on rank 1 does four loads. The code was > rank 0 rank 1 > _______ print x > x=val_1 print x > x=val_2 print x > _______ print x When I understand Bill Long correctly, then the 4 load(x) end up in > old_val > val_2 (because it was going through a very slow network path) > val_1 > val_2 wheras three enterleaved load instructions on rank 0 (i.e., in the same execution stream (thread) as the two stores) will always see old_val val_1 val_2 This is independent of compilers, because I only want to look at assembbler level. Bill Long, are you sure? I would expect that all loads go through the 1st Level Cache and as soon as it sees val_2 it should not be possible to see with a later issued instruction val_1. Best regards Rolf ----- Original Message ----- > From: "Bill Long" <longb(a)cray.com> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Tuesday, August 5, 2014 3:12:58 PM > Subject: Re: Short question on the ccNUMA memory reality > > Hi Rolf, > > > I assume you are expecting answers from people like Pavan and Bill G > for the MPI RMA perspective, and for the Fortran rules from me. (For > Fortran, map Rank 0 -> image 1 and Rank 1 -> Image 2). > > On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > > > Dear expert on ccNUMA, > > > > three questions, which hopefully may be trivial: > > > > 1. Question (sequential consistency on one location): > > ------------ > > > > Do I understand correctly that in the following patter > > on a shared Memory or a ccNUM shared memory > > I assume you mean what I think of as distributed memory here. > Otherwise, these are questions about OpenMP, and the rank 0 / rank > 1 separation does not make sense. > > > > > rank 0 rank 1 > > print x > > x=val_1 print x > > x=val_2 print x > > print x > > This depends on whether x is declared in a way that makes it > accessible from a remote rank. If not, then the code is illegal. So > I’ll assume it is accessible. > > Since there is no synchronization between rank 0 and rank 1, compiler > elimination of the x = val_1 assignment is allowed (and likely > expected). > > Even if the assignment is not eliminated, the print x on rank 1 > involves a “get” of x from rank 0. Assuming it is declared > volatile, to eliminate the “snap to local temp” optimization on rank > 1, it is still possible to get the values out of order. For > example, the print #2 could get routed through the network via some > extra long path, while print #3 could get routed on the fastest > path, resulting in the two “get” operations appearing at rank 0 in > an unpredictable order. > > [Note that by the Fortran rules, the code is not legal anyway. If > you define a variable on one rank and reference it (the print counts > as a reference) on another rank the execution segments containing > the definition and reference have to be ordered unless the > definition and reference are both explicitly atomic. Segment > ordering is done with synchronization statements.] > > > > > > the print statements can print only in the following > > sequence > > - some times the previous value > > - some times val_1 > > - and after some time val_2 and it then stays to print val_2 > > > > and that it can never be that a sequence with val_2 before val_1 > > can be produced, i.e., > > old_val > > val_2 > > val_1 > > val_2 > > is impossible. > > > > Also other values are impossible, e.g., some bit or byte-mix > > from val_1 and val_2. > > > On most hardware, assuming the values are “normal” size, such as > 64-bit, you will not get mixed bits. However, if you want to ensure > that, use the atomic get and put intrinsics. If the accesses had > been properly ordered by synchronizations, then you do not need the > atomics to ensure whole values. > > > > > > 2. Question: > > ----------- > > What is the largest size that the memory operations are atomic, > > i.e., that we do not see a bit or byte-mix from val_1 and val_2? > > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits > > into a cacheline? > > > Since the types in Fortran are parameterized, the processor supplies > a KIND value for the integer and logical types for which atomic > operations are supported. Typically kind corresponds to 64 bits, > but could be 32 on some architectures. > > > > > > 3. Question (about two updates): > > ----------- > > > > rank 0 rank 1 > > x=x_ld > > y=yold > > ---- necessary synchronizations ----- > > print x (which shows xold) > > print y (which shows yold) > > ---- necessary synchronizations ----- > > x=xnew > > y=ynew > > print x > > print y > > after some time > > print x > > print y > > > > Possible results are > > - xold,yold xold,yold xnew,ynew > > - xold,yold xnew,yold xnew,ynew > > - xold,yold xold,ynew xnew,ynew > > i.e., the y=ynew can arrive at another process > > faster than the x=xnew, although the storing > > process issues the stores in the sequence > > x=xnew, y=ynew. > > - xold,yold xnew,ynew xnew,ynew > > > > The assignments should represent the store instructions, > > and not the source code (because the compiler may modify > > sequence of instructions compared to the source code) > > > > Do I understand correctly, that the sequence of two > > store instructions two two different locations in one process > > may be visible at another process in a different sequence? > > Correct. As before, the “get” operations implied in the print > statements could arrive out of order. Also, while unlikely, the > compiler could decide to reverse the order of the assignments since > they are independent. (Assuming x and y cannot be aliased by, for > example, being pointers.) > > Cheers, > Bill > > > > > I ask all these questions to understand which memory model > > can be defined for MPI shared memory windows. > > > > Best regards > > Rolf > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > Bill Long > longb(a)cray.com > Fortran Technical Suport & voice: > 651-605-9024 > Bioinformatics Software Development fax: > 651-605-9142 > Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101 > > ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Bill Long" <longb(a)cray.com> > Sent: Tuesday, August 5, 2014 2:38:07 PM > Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality > > > Question 1 is hard to answer. First, as it came out in my output, > the first and last “print x” straddle the two ranks, so I’m not sure > what that means. But the following is valid for the compiler: > > > On rank 0. Since x = val_2 follows x = val_1 with no intervening use > of x, a good optimizing compiler may eliminate the store to x with > val_1 as unnecessary. > > > On rank 1, Since x is never assigned in this block, a compiler is > permitted to make a copy in a register (which would have the value x > had before this block was entered) and than print that value each > time print is called (e.g., it could build the call stack in > memory, and just pass that unchanged call stack to print for each > call). > > > And don’t forget - there’s a difference between when the print > completes on a process/thread and when the output appears - the > system could flush all print output from rank 1 before any print > output from rank 0 appears. > > > If x is volatile, then the compiler isn’t free to do the second (save > x and never load it again). > > > Bill > > > > > > > William Gropp > Director, Parallel Computing Institute Thomas M. Siebel Chair in > Computer Science > > > University of Illinois Urbana-Champaign > > > > > > > > On Aug 5, 2014, at 2:53 AM, Balaji, Pavan < balaji(a)anl.gov > wrote: > > > Huh? > > > > 1. Question (sequential consistency on one location): > ------------ > > Do I understand correctly that in the following patter > on a shared Memory or a ccNUM shared memory > > rank 0 rank 1 > print x > x=val_1 print x > x=val_2 print x > print x > > the print statements can print only in the following > sequence > - some times the previous value > - some times val_1 > - and after some time val_2 and it then stays to print val_2 > > and that it can never be that a sequence with val_2 before val_1 > can be produced, i.e., > old_val > val_2 > val_1 > val_2 > is impossible. > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by William Gropp 05 Aug '14

05 Aug '14

Question 1 is hard to answer. First, as it came out in my output, the first and last “print x” straddle the two ranks, so I’m not sure what that means. But the following is valid for the compiler: On rank 0. Since x = val_2 follows x = val_1 with no intervening use of x, a good optimizing compiler may eliminate the store to x with val_1 as unnecessary. On rank 1, Since x is never assigned in this block, a compiler is permitted to make a copy in a register (which would have the value x had before this block was entered) and than print that value each time print is called (e.g., it could build the call stack in memory, and just pass that unchanged call stack to print for each call). And don’t forget - there’s a difference between when the print completes on a process/thread and when the output appears - the system could flush all print output from rank 1 before any print output from rank 0 appears. If x is volatile, then the compiler isn’t free to do the second (save x and never load it again). Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Aug 5, 2014, at 2:53 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Huh? > >> 1. Question (sequential consistency on one location): >> ------------ >> >> Do I understand correctly that in the following patter >> on a shared Memory or a ccNUM shared memory >> >> rank 0 rank 1 >> print x >> x=val_1 print x >> x=val_2 print x >> print x >> >> the print statements can print only in the following >> sequence >> - some times the previous value >> - some times val_1 >> - and after some time val_2 and it then stays to print val_2 >> >> and that it can never be that a sequence with val_2 before val_1 >> can be produced, i.e., >> old_val >> val_2 >> val_1 >> val_2 >> is impossible. >

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

Pavan, please can you look at the more detailed question at end of question 1. Thank you for your answers on questions 2 and 3. Rolf ----- Original Message ----- > From: "Pavan Balaji" <balaji(a)anl.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Bill Long" <longb(a)cray.com> > Sent: Tuesday, August 5, 2014 9:53:06 AM > Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality > > > On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > > Dear expert on ccNUMA, > > Huh? > > > 1. Question (sequential consistency on one location): > > ------------ > > > > Do I understand correctly that in the following patter > > on a shared Memory or a ccNUM shared memory > > > > rank 0 rank 1 > > print x > > x=val_1 print x > > x=val_2 print x > > print x > > > > the print statements can print only in the following > > sequence > > - some times the previous value > > - some times val_1 > > - and after some time val_2 and it then stays to print val_2 > > > > and that it can never be that a sequence with val_2 before val_1 > > can be produced, i.e., > > old_val > > val_2 > > val_1 > > val_2 > > is impossible. > > When there are dependencies between statements, the compiler and > architecture don’t reorder them. > > > Also other values are impossible, e.g., some bit or byte-mix > > from val_1 and val_2. > > That is possible, unless you use some form of atomicity. In the case of automatic atomicity according to answer of 2nd question, together with the assumption that the compiler did not reorder (i.e. the stores are the store instructions and not source code lines), is it then guaranteed that the other process cannot see a different sequence, i.e., that it cannot see val_2 for a short time before it sees val_1, i.e., it is impossible that the other process would see > > old_val > > val_2 > > val_1 > > val_2 isn't it? > > 2. Question: > > ----------- > > What is the largest size that the memory operations are atomic, > > i.e., that we do not see a bit or byte-mix from val_1 and val_2? > > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits > > into a cacheline? > > It depends on the architecture, but all architectures I’m aware of do > give store atomicity of loads and stores till sizeof(void*), > assuming many things including the variables are volatile, aligned > correctly in memory, etc. Of course, this is not portable. To be > portable, you should use atomic operations, e.g., with OpenPA. > > > 3. Question (about two updates): > > ----------- > > > > rank 0 rank 1 > > x=x_ld > > y=yold > > ---- necessary synchronizations ----- > > print x (which shows xold) > > print y (which shows yold) > > ---- necessary synchronizations ----- > > x=xnew > > y=ynew > > print x > > print y > > after some time > > print x > > print y > > > > Possible results are > > - xold,yold xold,yold xnew,ynew > > - xold,yold xnew,yold xnew,ynew > > - xold,yold xold,ynew xnew,ynew > > i.e., the y=ynew can arrive at another process > > faster than the x=xnew, although the storing > > process issues the stores in the sequence > > x=xnew, y=ynew. > > - xold,yold xnew,ynew xnew,ynew > > > > The assignments should represent the store instructions, > > and not the source code (because the compiler may modify > > sequence of instructions compared to the source code) > > > > Do I understand correctly, that the sequence of two > > store instructions two two different locations in one process > > may be visible at another process in a different sequence? > > Correct. The compiler/hardware can reorder operations that they > perceive as unrelated. > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Balaji, Pavan 05 Aug '14

05 Aug '14

On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear expert on ccNUMA, Huh? > 1. Question (sequential consistency on one location): > ------------ > > Do I understand correctly that in the following patter > on a shared Memory or a ccNUM shared memory > > rank 0 rank 1 > print x > x=val_1 print x > x=val_2 print x > print x > > the print statements can print only in the following > sequence > - some times the previous value > - some times val_1 > - and after some time val_2 and it then stays to print val_2 > > and that it can never be that a sequence with val_2 before val_1 > can be produced, i.e., > old_val > val_2 > val_1 > val_2 > is impossible. When there are dependencies between statements, the compiler and architecture don’t reorder them. > Also other values are impossible, e.g., some bit or byte-mix > from val_1 and val_2. That is possible, unless you use some form of atomicity. > 2. Question: > ----------- > What is the largest size that the memory operations are atomic, > i.e., that we do not see a bit or byte-mix from val_1 and val_2? > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits > into a cacheline? It depends on the architecture, but all architectures I’m aware of do give store atomicity of loads and stores till sizeof(void*), assuming many things including the variables are volatile, aligned correctly in memory, etc. Of course, this is not portable. To be portable, you should use atomic operations, e.g., with OpenPA. > 3. Question (about two updates): > ----------- > > rank 0 rank 1 > x=x_ld > y=yold > ---- necessary synchronizations ----- > print x (which shows xold) > print y (which shows yold) > ---- necessary synchronizations ----- > x=xnew > y=ynew > print x > print y > after some time > print x > print y > > Possible results are > - xold,yold xold,yold xnew,ynew > - xold,yold xnew,yold xnew,ynew > - xold,yold xold,ynew xnew,ynew > i.e., the y=ynew can arrive at another process > faster than the x=xnew, although the storing > process issues the stores in the sequence > x=xnew, y=ynew. > - xold,yold xnew,ynew xnew,ynew > > The assignments should represent the store instructions, > and not the source code (because the compiler may modify > sequence of instructions compared to the source code) > > Do I understand correctly, that the sequence of two > store instructions two two different locations in one process > may be visible at another process in a different sequence? Correct. The compiler/hardware can reorder operations that they perceive as unrelated. -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Short question on the ccNUMA memory reality
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

Dear expert on ccNUMA, three questions, which hopefully may be trivial: 1. Question (sequential consistency on one location): ------------ Do I understand correctly that in the following patter on a shared Memory or a ccNUM shared memory rank 0 rank 1 print x x=val_1 print x x=val_2 print x print x the print statements can print only in the following sequence - some times the previous value - some times val_1 - and after some time val_2 and it then stays to print val_2 and that it can never be that a sequence with val_2 before val_1 can be produced, i.e., old_val val_2 val_1 val_2 is impossible. Also other values are impossible, e.g., some bit or byte-mix from val_1 and val_2. 2. Question: ----------- What is the largest size that the memory operations are atomic, i.e., that we do not see a bit or byte-mix from val_1 and val_2? Is it 1, 4, 8, 16 bytes or can it be a total struct that fits into a cacheline? 3. Question (about two updates): ----------- rank 0 rank 1 x=x_ld y=yold ---- necessary synchronizations ----- print x (which shows xold) print y (which shows yold) ---- necessary synchronizations ----- x=xnew y=ynew print x print y after some time print x print y Possible results are - xold,yold xold,yold xnew,ynew - xold,yold xnew,yold xnew,ynew - xold,yold xold,ynew xnew,ynew i.e., the y=ynew can arrive at another process faster than the x=xnew, although the storing process issues the stores in the sequence x=xnew, y=ynew. - xold,yold xnew,ynew xnew,ynew The assignments should represent the store instructions, and not the source code (because the compiler may modify sequence of instructions compared to the source code) Do I understand correctly, that the sequence of two store instructions two two different locations in one process may be visible at another process in a different sequence? I ask all these questions to understand which memory model can be defined for MPI shared memory windows. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [MPI Forum] #437: MPI 3.0 Errata: MPI Active Target synchronization requirements for shared memory, 2 alternatives
by Rolf Rabenseifner 04 Aug '14

04 Aug '14

Dear Bill, will you be at the Japan Forum meeting? in our telecon - provided I understood correctly - you told that you'll try to make a first proposal to solve the shared memory problems. Please take this email only as a possible input for your proposal, i.e., it is not thought as a discussion base. For me, it is important to finish this discussion more than two weeks before the Japan meeting, because it must be part of the MPI-3.0 errata and MPI-3.1. The two-week deadline is already Sep.1, 2014, i.e., 4 weeks from now! ----- Important sections that may need corrections: Sect.11.2.3 p409:16-17 mentions the first time "The locally allocated memory can be the target of load/store accesses by remote processes;" Here we may mention: The rules for RMA operation do not apply to these these remote load/store accesses; additional rules apply, see, e.g., Section 11.5.4A [the new one below] and 11.5.5 [on assertions] on pages ... and ... . p410:15-21: May need corrections p436:37-48: Especially the first sentence - #436 How is this section related to shared Memory. If there are additional rules for shared memory that do not apply to all unified memory Windows, then an additional section on "shared & unified" memory need to be added. I recommend to define it in the way "a load/store access in one process ... a load/store access in another process" and not in the way of "local" and "remote load/store". For me, after the MPI_Win_allocate_shared, there is not really memory portions that are local to the processes. The sentence p436:45-46 "The order in which data is written is not specified unless further synchronization is used." If a process issues several load and stores to the same location, they are executed in the given sequence without further need of synchronization rank 0 rank 1 x=1 print x x=2 print x If two processes access the same location, may it possible that the three prints may be other than 1-1 1-2 2-2? Do we want to keep this undefined? I would propose to add a rationale like --- Rationale. If two processes on a ccNUMA node access the same memory location with an intermediate process-to-process synchronization then the outcome is still undefined because the first memory operation need not to be finished before the second memory operation starts. A well-defined execution may require the following sequence: 1) the memory access on the first process; 2) a local memory fence to guarantee that the memory operation is finished; 3) the process-to-process synchronization (e.g. send/recv) to inform the second process; 4) a local memory fence on the second process to guarantee that subsequent memory operations are issued to the memory 5) the memory operation on the second process. To guarantee the outcome of memory accesses in combination with the synchronization as described in Section 11.5.4A on page ... (NEW SECTION, SEE BELOW), the synchronization operation will issue such local memory fences if needed. End of rationale. --- This rationale makes it understandable, what we discussed in such many emails about the need of MPI_WIN_SYNC. The major part are the new synchronization rules: I propose to put them together in one new section after 11.5.4 11.5.4A Shared Memory Synchronization In the case of an MPI shared memory window additional rules apply for synchronizing load and store accesses from several processes to the same location. A location must be at least a byte. In the following patterns, locations in a shared memory window are noted with variable names A, B, C, loads from such windows are noted with load(...) and stores are noted by assignments to these variables. Patterns with active target communication and MPI_Win_sync: process P0 process P1 A=val_1 Sync-to-P1 --> Sync-from-P0 load(A) load(B) Sync-to-P1 --> Sync-from-P0 B=val_2 C=val_3 Sync-to-P1 --> Sync-from-P0 C=val_4 load (C) with Sync-to-P1 --> Sync-from-P0 can be 1. MPI_Win_fence --> MPI_Win_fence 1) 2. MPI_Win_post --> MPI_Win_start 2) 3. MPI_Win_complete--> MPI_Win_wait 3) 4. MPI_Win_sync any process sync to P1 --> any process sync from P0 4) MPI_Win_sync Footnotes: 1) MPI_Win_fence synchronizes in both directions and between every process in the process group of the window. 2) The arrow means that P1 is in the origin group passed to MPI_Win_post in P0, and that P0 is in the target group passed to MPI_Win_start. Additional calls to MPI_Win_complete (in P1 after MPI_Win_start) and MPI_Win_wait (in P0 after MPI_Win_post) are need. The location of these calls do not influence the guaranteed outcome rules. 3) The arrow means that P1 is in the target group passed to MPI_Win_start that corresponds to MPI_Win_complete in P0, and P0 is in the origin group passed to MPI_Win_post that corresponds to MPI_Win_wait in P1. Additional calls to MPI_Win_start (in P0 before MPI_Win_complete) and MPI_Win_post (in P1 before MPI_Win_wait) are need. The location of these calls do not influence the guaranteed outcome rules. 4) The synchronization may be done with methods from MPI (e.g. send-->recv) or with other methods. The requirements for using MPI_Win_sync (e.g. within a passive target epoch which may be provided with MPI_Win_lock_all) are not shown in thee pattern Patterns with lock/unlock synchronization: Within passive target communication, two locks L1 and L2 may be scheduled L1 before L2 or L2 before L1. In the following patterns, the arrow means that the lock in P0 was scheduled before the lock in P1. For the following load/store/sync patterns process P0 process P1 MPI_Win_lock exclusive A=val_1 MPI_Win_Unlock --> MPI_Lock shared or exclusive load(A) MPI_Win_lock MPI_Win_lock shared or exclusive load(B) MPI_Win_Unlock --> MPI_Lock exclusive B=val_2 MPI_Win_lock MPI_Win_lock exclusive C=val_3 MPI_Win_Unlock --> MPI_Lock exclusive C=val_4 load (C) MPI_Win_lock Note that each rank of a window is connected to a separate lock. In a shared window, these locks are not connected to specific memory portions of the shared window, i.e., each lock can be used to protect any portion of a shared memory window. In the patterns above, it is guaranteed - that the load(A) in P1 loads val_1 (this is the Write-read-rule), - that the load(B) in P0 is not affected by the store of val_2 in P1 (read-write-rule), and - that the load(C) in P1 loads val_4 (write-write-rule). ---- I do not understand which rules are guaranteed for - MPI_Win_flush (and MPI_Win_flush_all) - MPI_Win_flush_local (and MPI_Win_flush_local_all) I hope that somebody understands these calls and can produce the related patterns. ---- About Sect. 11.5.5 Assertions - MPI_MODE_NOSTORE: I expect that the proposal in #429 is not helpful: it enlarges the hint about "not updated by stores since last synchronization" from local stores to all stores by the whole process group. The reason for this hint is to prevent "cache synchronization". This cache synchronization is local and therefore the remote stores do not count. The new proposal is simple: p452:1 and p452.9: "stores" --> "local stores" Additionally, I would add the following sentence: "In the case of a shared memory window, such local stores can be issued to any portion of the shared memory window." Reason: Nobody should think that "local" means "only to the window portion that was defined in the local MPI_WIN_ALLOCATE_SHARED". Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0