August 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jim Dinan 29 Aug '13

29 Aug '13

Here's a refinement of the earlier draft erratum that incorporates this change: A window synchronization (e.g. MPI_WIN_SYNC, MPI_WIN_LOCK/UNLOCK, etc.) can be used to order load/store operations with respect to actions performed by other processes in the target's window. * In addition, an MPI implementation may also support the use of external operations to provide this ordering.* If no synchronization is performed, the ordering of load/store operations with respect to actions performed by other processes in the target's window is not defined. Rationale: The local process' view of the window may not be consistent with the window because of performance optimizations or the consistency model provided by the system. ~Jim. On Thu, Aug 29, 2013 at 3:17 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > The only difference would be that we'll add there might be other > implementation-specific ways to do the same. > > -- Pavan > > On Aug 29, 2013, at 10:30 AM, Barrett, Brian W wrote: > > > On 8/28/13 7:22 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > > > >> > >> On 08/28/2013 03:47 PM, Barrett, Brian W wrote: > >>> And, further, I believe > >>> that: > >>> > >>> X = 0 > >>> Win_lock_all > >>> Recv > >>> do { compiler_fence(); } while (0 == X); > >>> > >>> will eventually complete on any platform that supports UNIFIED, as the > >>> public and private copies are eventually consistent. This gives us the > >>> behaviors necessary for implementing SHMEM on MPI-3. > >> > >> Right, good point. We could retain the definition that the public and > >> private windows will eventually be the same without additional MPI > calls. > >> > >> With that in mind, does requiring WIN_SYNC for immediate synchronization > >> of the two windows + a statement that there might be > >> implementation-specific ways (e.g., memory barriers) to do the same > >> sound reasonable from an OpenSHMEM point-of-view? > > > > To me, yes. > > > > Brian > > > > -- > > Brian W. Barrett > > Scalable System Software Group > > Sandia National Laboratories > > > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 29 Aug '13

29 Aug '13

The only difference would be that we'll add there might be other implementation-specific ways to do the same. -- Pavan On Aug 29, 2013, at 10:30 AM, Barrett, Brian W wrote: > On 8/28/13 7:22 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > >> >> On 08/28/2013 03:47 PM, Barrett, Brian W wrote: >>> And, further, I believe >>> that: >>> >>> X = 0 >>> Win_lock_all >>> Recv >>> do { compiler_fence(); } while (0 == X); >>> >>> will eventually complete on any platform that supports UNIFIED, as the >>> public and private copies are eventually consistent. This gives us the >>> behaviors necessary for implementing SHMEM on MPI-3. >> >> Right, good point. We could retain the definition that the public and >> private windows will eventually be the same without additional MPI calls. >> >> With that in mind, does requiring WIN_SYNC for immediate synchronization >> of the two windows + a statement that there might be >> implementation-specific ways (e.g., memory barriers) to do the same >> sound reasonable from an OpenSHMEM point-of-view? > > To me, yes. > > Brian > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jim Dinan 29 Aug '13

29 Aug '13

On Thu, Aug 29, 2013 at 11:30 AM, Barrett, Brian W <bwbarre(a)sandia.gov>wrote: > On 8/28/13 7:22 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > > > > >On 08/28/2013 03:47 PM, Barrett, Brian W wrote: > >> And, further, I believe > >> that: > >> > >> X = 0 > >> Win_lock_all > >> Recv > >> do { compiler_fence(); } while (0 == X); > >> > >> will eventually complete on any platform that supports UNIFIED, as the > >> public and private copies are eventually consistent. This gives us the > >> behaviors necessary for implementing SHMEM on MPI-3. > > > >Right, good point. We could retain the definition that the public and > >private windows will eventually be the same without additional MPI calls. > > > >With that in mind, does requiring WIN_SYNC for immediate synchronization > >of the two windows + a statement that there might be > >implementation-specific ways (e.g., memory barriers) to do the same > >sound reasonable from an OpenSHMEM point-of-view? > > To me, yes. Isn't this the same semantic that we were already leaning toward? >From Aug. 6: ---8<--- I prefer option #1 -- that a window synchronization (e.g. Win_sync) can be used to order load/store operations with respect to actions performed by other processes in the target's window. If no ordering is enforced, the MPI standard does not define what is seen by load operations at the target process. As a rationale, the local process' view of the window may not be consistent with the window because of performance optimizations or the consistency model of the underlying architecture. This would allow e.g. SHMEM implementations to still use MPI-3 RMA, but they would have to rely on a behavior that is defined by the architecture/implementation, as they currently do. ---8<--- We never discussed removing the eventually synchronized semantic for unified windows. The only change was that MPI_WIN_SYNC should be used when the user wants "eventually" to be "now". If the platform ensures ordering through some other means, or a platform-specific non-MPI operation is used to order operations, that's fine, you are just relying on an extension for your platform. ~Jim.

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jim Dinan 29 Aug '13

29 Aug '13

On Wed, Aug 28, 2013 at 4:47 PM, Barrett, Brian W <bwbarre(a)sandia.gov>wrote: > >>> Also, I think we are ignoring the point Jim raised: does OpenSHMEM > >>> really need these semantics? > >> > >> I think this is a grey area of OpenSHMEM and my belief is that it really > >> does need those semantics. > > > >Perhaps this is just an oversight in OpenSHMEM and can be fixed? > > I don't think it's an oversight. Remember that OpenSHMEM, while a great > leap forward from SGI's man pages, is still a fairly loose specification. > There are definitely codes that spin on a completion word (or poll and > move on to other things, where shmem_wait wouldn't work) in the wild, and > they're considered correct codes. What does this spinning look like? Are any platform-specific operations used? ~Jim.

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Barrett, Brian W 29 Aug '13

29 Aug '13

On 8/28/13 7:22 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > >On 08/28/2013 03:47 PM, Barrett, Brian W wrote: >> And, further, I believe >> that: >> >> X = 0 >> Win_lock_all >> Recv >> do { compiler_fence(); } while (0 == X); >> >> will eventually complete on any platform that supports UNIFIED, as the >> public and private copies are eventually consistent. This gives us the >> behaviors necessary for implementing SHMEM on MPI-3. > >Right, good point. We could retain the definition that the public and >private windows will eventually be the same without additional MPI calls. > >With that in mind, does requiring WIN_SYNC for immediate synchronization >of the two windows + a statement that there might be >implementation-specific ways (e.g., memory barriers) to do the same >sound reasonable from an OpenSHMEM point-of-view? To me, yes. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 28 Aug '13

28 Aug '13

On 08/28/2013 03:47 PM, Barrett, Brian W wrote: > And, further, I believe > that: > > X = 0 > Win_lock_all > Recv > do { compiler_fence(); } while (0 == X); > > will eventually complete on any platform that supports UNIFIED, as the > public and private copies are eventually consistent. This gives us the > behaviors necessary for implementing SHMEM on MPI-3. Right, good point. We could retain the definition that the public and private windows will eventually be the same without additional MPI calls. With that in mind, does requiring WIN_SYNC for immediate synchronization of the two windows + a statement that there might be implementation-specific ways (e.g., memory barriers) to do the same sound reasonable from an OpenSHMEM point-of-view? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Underwood, Keith D 28 Aug '13

28 Aug '13

> >>> Also, I think we are ignoring the point Jim raised: does OpenSHMEM > >>> really need these semantics? > >> > >> I think this is a grey area of OpenSHMEM and my belief is that it > >> really does need those semantics. > > > >Perhaps this is just an oversight in OpenSHMEM and can be fixed? > > I don't think it's an oversight. Remember that OpenSHMEM, while a great > leap forward from SGI's man pages, is still a fairly loose specification. > There are definitely codes that spin on a completion word (or poll and move > on to other things, where shmem_wait wouldn't work) in the wild, and > they're considered correct codes. And, I think they *should* be correct. It should be possible in OpenSHMEM and MPI3-RMA-UNIFIED to do shared memory programming using threads locally in the symmetric heap/unified window and have the same mechanisms "just work" when updates from remote accesses arrive. This provides for an "easy" to access programming model while providing explicit remote memory accesses.

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Barrett, Brian W 28 Aug '13

28 Aug '13

On 8/28/13 3:37 AM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: >On 08/27/2013 10:35 PM, Barrett, Brian W wrote: >> Perhaps implementation-defined is the wrong word. I'm not sure I agree >> with keith's undefined fright, but perhaps platform-defined? There's a >> big difference between a replacement for MPI_WAIT and a memory barrier >>in >> a loop. > >The comparison here should be a platform-specific memory barrier in a >loop vs. MPI_WIN_SYNC in a loop. > >My concern with "platform-defined" is that we are assuming that there is >exactly one way of doing this for a given platform, which is known to >the MPI implementation and to the user. That is mandating something on >the implementation and makes me uncomfortable. I see that concern, and agree with it. >"Implementation-defined" is better, in that the implementation is free >to do whatever it wants. The user can do the same thing if she knows >what the implementation does. But practically, there's likely no >difference implementation-defined and undefined. My problem with implementation defined is exactly your problem with platform defined. But since I think both are better than "undefined", I'm not sure I care all that much :). >Can you also clarify which of the two items below is the real concern >here (or both?): > >1. Is it the cost of the MPI_WIN_SYNC function call? If yes, how about >we deprecate/remove PMPI_WIN_SYNC? That way, MPI_WIN_SYNC can be a >macro and truly a no-op on some platforms, so the concern that it might >be expensive will go away? That's certainly one concern, but the minor concern. >2. Is it the fact that I have to do some additional call even on >platforms where it is not needed (e.g., x86)? So basically you want to >do this: > >P0: >Win_lock_all >Put(X) >Flush >Send > >P1: >Win_lock_all >Recv >/* No Win_sync here */ >read X I believe on x86 that your example is correct. I think you either need a memory barrier or WIN_SYNC on other platforms. And, further, I believe that: X = 0 Win_lock_all Recv do { compiler_fence(); } while (0 == X); will eventually complete on any platform that supports UNIFIED, as the public and private copies are eventually consistent. This gives us the behaviors necessary for implementing SHMEM on MPI-3. >>> Also, I think we are ignoring the point Jim raised: does OpenSHMEM >>> really need these semantics? >> >> I think this is a grey area of OpenSHMEM and my belief is that it really >> does need those semantics. > >Perhaps this is just an oversight in OpenSHMEM and can be fixed? I don't think it's an oversight. Remember that OpenSHMEM, while a great leap forward from SGI's man pages, is still a fairly loose specification. There are definitely codes that spin on a completion word (or poll and move on to other things, where shmem_wait wouldn't work) in the wild, and they're considered correct codes. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 28 Aug '13

28 Aug '13

Brian, On 08/27/2013 10:35 PM, Barrett, Brian W wrote: > Perhaps implementation-defined is the wrong word. I'm not sure I agree > with keith's undefined fright, but perhaps platform-defined? There's a > big difference between a replacement for MPI_WAIT and a memory barrier in > a loop. The comparison here should be a platform-specific memory barrier in a loop vs. MPI_WIN_SYNC in a loop. My concern with "platform-defined" is that we are assuming that there is exactly one way of doing this for a given platform, which is known to the MPI implementation and to the user. That is mandating something on the implementation and makes me uncomfortable. "Implementation-defined" is better, in that the implementation is free to do whatever it wants. The user can do the same thing if she knows what the implementation does. But practically, there's likely no difference implementation-defined and undefined. Can you also clarify which of the two items below is the real concern here (or both?): 1. Is it the cost of the MPI_WIN_SYNC function call? If yes, how about we deprecate/remove PMPI_WIN_SYNC? That way, MPI_WIN_SYNC can be a macro and truly a no-op on some platforms, so the concern that it might be expensive will go away? 2. Is it the fact that I have to do some additional call even on platforms where it is not needed (e.g., x86)? So basically you want to do this: P0: Win_lock_all Put(X) Flush Send P1: Win_lock_all Recv /* No Win_sync here */ read X >> Also, I think we are ignoring the point Jim raised: does OpenSHMEM >> really need these semantics? > > I think this is a grey area of OpenSHMEM and my belief is that it really > does need those semantics. Perhaps this is just an oversight in OpenSHMEM and can be fixed? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Barrett, Brian W 27 Aug '13

27 Aug '13

On 8/27/13 1:59 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: >On 08/27/2013 01:49 PM, Barrett, Brian W wrote: >> I'm having trouble with the exact phrasing, but my thought is that we >> should have an advice to users that says 1) while the view of memory is >> consistent in UNIFIED, other artifacts (such as processor or memory >> controller reordering) can result in unexpected behavior 2) MPI_WIN_SYNC >> may be used to provide stronger ordering guarantees and 3) an >>application >> programmer may use other, implementation defined, semantics to provide >> required ordering guarantees. > >Isn't (3) true for any MPI call in general, since the user can use >implementation-defined alternate mechanisms for any of them? Perhaps implementation-defined is the wrong word. I'm not sure I agree with keith's undefined fright, but perhaps platform-defined? There's a big difference between a replacement for MPI_WAIT and a memory barrier in a loop. >Also, I think we are ignoring the point Jim raised: does OpenSHMEM >really need these semantics? I think this is a grey area of OpenSHMEM and my belief is that it really does need those semantics. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0