July 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jed Brown 30 Jul '13

30 Jul '13

Pavan Balaji <balaji(a)mcs.anl.gov> writes: > This is what I said is the disagreement in the WG. I can pull up the > old email chain if needed, but I think others can too. One side was > arguing that there's no such guarantee and you need to do a WIN_SYNC to > see the value. The other side was arguing that the WIN_SYNC should not > be needed; FLUSH + SEND on the origin should be enough. Hmm, linux/Documentation/memory-barriers.txt says: CACHE COHERENCY VS DMA ---------------------- Not all systems maintain cache coherency with respect to devices doing DMA. In such cases, a device attempting DMA may obtain stale data from RAM because dirty cache lines may be resident in the caches of various CPUs, and may not have been written back to RAM yet. To deal with this, the appropriate part of the kernel must flush the overlapping bits of cache on each CPU (and maybe invalidate them as well). In addition, the data DMA'd to RAM by a device may be overwritten by dirty cache lines being written back to RAM from a CPU's cache after the device has installed its own data, or cache lines present in the CPU's cache may simply obscure the fact that RAM has been updated, until at such time as the cacheline is discarded from the CPU's cache and reloaded. To deal with this, the appropriate part of the kernel must invalidate the overlapping bits of the cache on each CPU. I've taken this to mean that you can't guarantee that a DMA write will "eventually" be visible to the CPU (because the cache line could hang around arbitrarily long). Are implementations doing something here to ensure that cache on the target is invalidated after RDMA operations? I think (though not with confidence) that CPU cache invalidation will eventually (with a practical upper bound) propagate to other CPUs. Comparing to Alpha with split cache lines, one of the buses could be busy and thus not update despite proper memory ordering on the write end. The implication is that each cache has a (fair?) queue and it cannot be arbitrarily long, though I've never seen a statement providing an upper bound on how long the cache bank could be busy.

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Sur, Sayantan 30 Jul '13

30 Jul '13

> On 07/30/2013 10:28 AM, Jim Dinan wrote: > > I believe this is adequately specified on pg 436, line 37. P1 will > > *eventually* see the new value for "a" without any additional > > synchronization operations, but neither the flush by P0 nor the Recv > > by > > P1 guarantee that P1 will see the new value immediately. > > This is what I said is the disagreement in the WG. I can pull up the old email > chain if needed, but I think others can too. One side was arguing that there's > no such guarantee and you need to do a WIN_SYNC to see the value. The > other side was arguing that the WIN_SYNC should not be needed; FLUSH + > SEND on the origin should be enough. > Here's the old thread: http://lists.mpi-forum.org/mpi3-rma/2011/03/0533.php Looks like the idea to call MPI_Win_sync for Unified got votes from both Bill and Torsten. Were there others who were in this camp? Thanks, Sayantan.

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 30 Jul '13

30 Jul '13

On 07/30/2013 10:28 AM, Jim Dinan wrote: > I believe this is adequately specified on pg 436, line 37. P1 will > *eventually* see the new value for "a" without any additional > synchronization operations, but neither the flush by P0 nor the Recv by > P1 guarantee that P1 will see the new value immediately. This is what I said is the disagreement in the WG. I can pull up the old email chain if needed, but I think others can too. One side was arguing that there's no such guarantee and you need to do a WIN_SYNC to see the value. The other side was arguing that the WIN_SYNC should not be needed; FLUSH + SEND on the origin should be enough. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jeff Hammond 30 Jul '13

30 Jul '13

Yeah, all I ever wanted out of unified was eventual asynchronous arrival but when I was thinking about it in detail, I couldn't figure out how I could ever make use of this in a way that wouldn't look exactly like SEPARATE. Jeff On Tue, Jul 30, 2013 at 10:28 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi Pavan, > > I believe this is adequately specified on pg 436, line 37. P1 will > *eventually* see the new value for "a" without any additional > synchronization operations, but neither the flush by P0 nor the Recv by P1 > guarantee that P1 will see the new value immediately. > > ~Jim. > > > On Sat, Jul 27, 2013 at 1:08 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >> Dear all, >> >> We had some discussion on the semantics of UNIFIED in the past, that >> we were hoping to have clarified in MPI-3.1 or the errata. This email >> is to revive that discussion, so that we don't forget to address it. >> >> Specifically, the concern was that some members of the WG believed >> that in the UNIFIED model, data is usable by the remote process after >> a PUT without an additional WIN_SYNC, while some members believed that >> it is not. Here's the example in question: >> >> P0: >> Win_lock_all >> Put(a, P1) >> Flush >> MPI_Send(P1) >> >> P1: >> Win_lock_all >> MPI_Recv(P0) >> read a >> >> The question was whether the above program was valid without a >> WIN_SYNC on P1 between the Recv(P0) and "read a". If we want this to >> be valid in the UNIFIED model, only x86-like architectures can provide >> UNIFIED efficiently. Other architectures, such as PPC or ARM, that >> require an additional read barrier on P1 will not be able to provide >> UNIFIED even if they are cache-coherent, unless they add a memory barrier >> in every other MPI call (e.g., MPI_Recv in this case). >> >> One possible solution we discussed was to clarify that this is not >> allowed in UNIFIED, but provide a third memory model called >> UBER_SUPER_UNIFIED, that will allow this. (or say that it is allowed >> in UNIFIED and provide a third model called KIND_OF_UNIFIED, which is >> in between UNIFIED and SEPARATE). >> >> Other solutions are welcome. >> >> Irrespective of when we make the change of possibly adding an >> additional memory model (MPI-3.1, MPI-4, whatever), we should clarify >> the standard on what is allowed in MPI-3 and what is not, as an errata >> item. Without that, it's confusing for implementors. >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jim Dinan 30 Jul '13

30 Jul '13

Hi Pavan, I believe this is adequately specified on pg 436, line 37. P1 will *eventually* see the new value for "a" without any additional synchronization operations, but neither the flush by P0 nor the Recv by P1 guarantee that P1 will see the new value immediately. ~Jim. On Sat, Jul 27, 2013 at 1:08 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Dear all, > > We had some discussion on the semantics of UNIFIED in the past, that > we were hoping to have clarified in MPI-3.1 or the errata. This email > is to revive that discussion, so that we don't forget to address it. > > Specifically, the concern was that some members of the WG believed > that in the UNIFIED model, data is usable by the remote process after > a PUT without an additional WIN_SYNC, while some members believed that > it is not. Here's the example in question: > > P0: > Win_lock_all > Put(a, P1) > Flush > MPI_Send(P1) > > P1: > Win_lock_all > MPI_Recv(P0) > read a > > The question was whether the above program was valid without a > WIN_SYNC on P1 between the Recv(P0) and "read a". If we want this to > be valid in the UNIFIED model, only x86-like architectures can provide > UNIFIED efficiently. Other architectures, such as PPC or ARM, that > require an additional read barrier on P1 will not be able to provide > UNIFIED even if they are cache-coherent, unless they add a memory barrier > in every other MPI call (e.g., MPI_Recv in this case). > > One possible solution we discussed was to clarify that this is not > allowed in UNIFIED, but provide a third memory model called > UBER_SUPER_UNIFIED, that will allow this. (or say that it is allowed > in UNIFIED and provide a third model called KIND_OF_UNIFIED, which is > in between UNIFIED and SEPARATE). > > Other solutions are welcome. > > Irrespective of when we make the change of possibly adding an > additional memory model (MPI-3.1, MPI-4, whatever), we should clarify > the standard on what is allowed in MPI-3 and what is not, as an errata > item. Without that, it's confusing for implementors. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > ______________________________**_________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/**mailman/listinfo.cgi/mpi3-rma<http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma> >

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jed Brown 30 Jul '13

30 Jul '13

Pavan Balaji <balaji(a)mcs.anl.gov> writes: >> Of course, while(flag) will be optimized away by a lot of compilers >> without some magic annotation that would prevent the read from >> crossing it... > > I can't get any compiler to optimize it away. What did you try? Gcc and clang both elide the conditional from the loop body. a.c: extern void bar(int*); int foo() { int flag; bar(&flag); while (flag); return 1; } $ gcc -Os -c -S a.c -o - # infinite loop at .L4 .file "a.c" .text .globl foo .type foo, @function foo: .LFB0: .cfi_startproc subq $24, %rsp .cfi_def_cfa_offset 32 leaq 12(%rsp), %rdi call bar cmpl $0, 12(%rsp) je .L5 .L4: jmp .L4 .L5: movl $1, %eax addq $24, %rsp .cfi_def_cfa_offset 8 ret .cfi_endproc .LFE0: .size foo, .-foo .ident "GCC: (GNU) 4.8.1" .section .note.GNU-stack,"",@progbits $ clang -Os -c -S a.c -o - # infinite loop at .LBB0_1 .file "a.c" .text .globl foo .type foo,@function foo: # @foo .cfi_startproc # BB#0: pushq %rax .Ltmp1: .cfi_def_cfa_offset 16 leaq 4(%rsp), %rdi callq bar cmpl $0, 4(%rsp) je .LBB0_2 .LBB0_1: # %..split_crit_edge # =>This Inner Loop Header: Depth=1 jmp .LBB0_1 .LBB0_2: # %.split1 movl $1, %eax popq %rdx ret .Ltmp2: .size foo, .Ltmp2-foo .cfi_endproc .section ".note.GNU-stack","",@progbits

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 29 Jul '13

29 Jul '13

On 07/29/2013 11:21 PM, Underwood, Keith D wrote: >> On 07/29/2013 10:58 PM, Underwood, Keith D wrote: >> Are you saying that the architecture cannot track that these two are >> nonoverlapping buffers? > > Exactly. MPI_Recv takes a void* and a length. Because MPI_recv > knows that the architecture is not parsing the arguments and > semantics of MPI_Recv, it must memory barrier (if that is an issue > for the implementation). The compiler can't track this, but the architecture should be able to if it's looking at bytes accessed. >> P0: >> Barrier >> Win_lock_all(win1) >> Put(a, P1) >> Flush >> flag = 1; >> Put(flag, P1) >> >> P1: >> flag = 0; >> Barrier >> Win_lock_all >> while (flag); >> read a > > Now, this is an interesting example. Do we promise that this will work? We say that the once the flag turns 1, it'll stay 1. The origin guarantees that the flag only turns 1 after the data 'a' is visible at the target process for load/store. > Of course, while(flag) will be optimized away by a lot of compilers > without some magic annotation that would prevent the read from > crossing it... I can't get any compiler to optimize it away. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Underwood, Keith D 29 Jul '13

29 Jul '13

> On 07/29/2013 10:58 PM, Underwood, Keith D wrote: > >> P1: > >> Win_lock_all > >> MPI_Recv(P0, flag) > >> read a > > > > Can you point to the line in there that tells the architecture that > > the read of A is not touched by MPI_Recv? Let's say the data movement > > for MPI_Recv is done by a NIC or done by another core. How can the > > microarchitecture tell the difference? > > 'flag' and 'a' are two different buffers, so presumably reordering MPI_RECV > and 'read a' should be permitted. > > Are you saying that the architecture cannot track that these two are > nonoverlapping buffers? Exactly. MPI_Recv takes a void* and a length. Because MPI_recv knows that the architecture is not parsing the arguments and semantics of MPI_Recv, it must memory barrier (if that is an issue for the implementation). > If it's the latter, then let's consider the following new example: > > P0: > Barrier > Win_lock_all(win1) > Put(a, P1) > Flush > flag = 1; > Put(flag, P1) > > P1: > flag = 0; > Barrier > Win_lock_all > while (flag); > read a Now, this is an interesting example. Do we promise that this will work? Of course, while(flag) will be optimized away by a lot of compilers without some magic annotation that would prevent the read from crossing it...

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 29 Jul '13

29 Jul '13

On 07/29/2013 10:58 PM, Underwood, Keith D wrote: >> P1: >> Win_lock_all >> MPI_Recv(P0, flag) >> read a > > Can you point to the line in there that tells the architecture that > the read of A is not touched by MPI_Recv? Let's say the data > movement for MPI_Recv is done by a NIC or done by another core. How > can the microarchitecture tell the difference? 'flag' and 'a' are two different buffers, so presumably reordering MPI_RECV and 'read a' should be permitted. Are you saying that the architecture cannot track that these two are nonoverlapping buffers? Or are you saying that the "poll completion queue" equivalent for the network receive should already be doing a memory barrier for a correct network stack anyway? If it's the latter, then let's consider the following new example: P0: Barrier Win_lock_all(win1) Put(a, P1) Flush flag = 1; Put(flag, P1) P1: flag = 0; Barrier Win_lock_all while (flag); read a Remember MPI-3's nasty semantics of single-byte flags turning to non-zero and staying there :-). -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Underwood, Keith D 29 Jul '13

29 Jul '13

> On 07/29/2013 10:48 PM, Underwood, Keith D wrote: > >> On 07/29/2013 10:18 PM, Underwood, Keith D wrote: > >>> And, um, technically, if loads can be hoisted across the MPI_Recv, > >>> then the MPI_Recv semantics are incorrect and you have to have a > >>> memory barrier in MPI_Recv. > >> > >> Why is reordering not permitted for two unrelated memory loads? > >> AFAIK, ARM allows that. > > > > I'm talking about a load of the data written for the MPI_Recv getting > > hoisted across MPI_Recv. There is no guarantee of anything in that > > MPI_Recv actually touching the data being received. There is no > > guarantee of anything telling the microarchitecture that you are doing > > an illegal hoist across that receive. > > No, we are obviously not talking about data reordering for dependent data. > In the example I gave, the recv of some other data is essentially a flag for me > to tell that the origin has flushed some data to the target. Here's the > example again: > > P0: > Win_lock_all > Put(a, P1) > Flush > MPI_Send(P1, flag) > > P1: > Win_lock_all > MPI_Recv(P0, flag) > read a > Can you point to the line in there that tells the architecture that the read of A is not touched by MPI_Recv? Let's say the data movement for MPI_Recv is done by a NIC or done by another core. How can the microarchitecture tell the difference?

1 0