August 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model clarification
by Pavan Balaji 04 Aug '13

04 Aug '13

On 08/04/2013 06:10 PM, Barrett, Brian W wrote: > I think we have to agree to disagree on that one, in that I really don't > think your example is a it's a common usage model worth destroying > UNIFIED over. Most one-sided codes I've seen use one-sided ops for > everything in the window, but I haven't seen as many codes as I might like. OK. FWIW, one case where this became a problem was when we were using PUT/GET accesses were issued from window memory to window memory. Even for PUT/GET accesses, the origin accesses are considered load/store accesses; only the target accesses are considered RMA accesses. One of the applications that ran into this was our bio assembly application. Another was some secretive application Bill's student is working on, which I don't know all the details of (I can get more details, depending on how much she's permitted to say, if there is interest). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model clarification
by Jed Brown 04 Aug '13

04 Aug '13

Pavan Balaji <balaji(a)mcs.anl.gov> writes: > SEPARATE is too expensive for such cases. What is the lightest weight way for the source to create an ordering at the target (which will be using MPI_Win_sync)? How does the overhead of that compare to MPI_Win_sync?

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model clarification
by Barrett, Brian W 04 Aug '13

04 Aug '13

On 8/4/13 5:02 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov<mailto:[email protected]>> wrote: On 08/04/2013 05:56 PM, Barrett, Brian W wrote: So, I guess I really don't see a problem with what we have today. As soon as you require a target side WIN_SYNC, there's no point in using Unified; the user and the implementation can live quite happily with SEPARATE. And, I believe, make SEPARATE go fast on platforms that only need a memory barrier to ensure target side ordering, rather than a complicated cache protocol. That's not true. If two processes are accessing nonoverlapping memory locations (PUT/GETs and load/stores), it would still be very useful. With SEPARATE additional synchronization would be required for such accesses. UNIFIED is too complex if the user mostly cares about nonoverlapping memory regions, and doesn't mind paying a small penalty for overlapping regions. SEPARATE is too expensive for such cases. I think we have to agree to disagree on that one, in that I really don't think your example is a it's a common usage model worth destroying UNIFIED over. Most one-sided codes I've seen use one-sided ops for everything in the window, but I haven't seen as many codes as I might like. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model clarification
by Pavan Balaji 04 Aug '13

04 Aug '13

Brian, On 08/04/2013 05:56 PM, Barrett, Brian W wrote: > So, I guess I really don't see a problem with what we have today. As > soon as you require a target side WIN_SYNC, there's no point in using > Unified; the user and the implementation can live quite happily with > SEPARATE. And, I believe, make SEPARATE go fast on platforms that only > need a memory barrier to ensure target side ordering, rather than a > complicated cache protocol. That's not true. If two processes are accessing nonoverlapping memory locations (PUT/GETs and load/stores), it would still be very useful. With SEPARATE additional synchronization would be required for such accesses. UNIFIED is too complex if the user mostly cares about nonoverlapping memory regions, and doesn't mind paying a small penalty for overlapping regions. SEPARATE is too expensive for such cases. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model clarification
by Barrett, Brian W 04 Aug '13

04 Aug '13

On 8/2/13 7:29 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov<mailto:[email protected]>> wrote: On 08/02/2013 08:27 PM, Underwood, Keith D wrote: That was specifically *not* the point of unified. The point of unified was that "if the platform can support it, the user can get something better". Remember all those conversations about synchronization using hands waving across the machine room? Unless you guys want to go get rid of UNIFIED? There really isn't a point to unified if you have to call MPI_Win_sync, is there? Yes, I remember that discussion. However, what I took away from that discussion was that the user *could* do something outside of MPI, not *must*, to ensure correct behavior. If that was not the intention, then there needs to be a memory model where that's the case. I took away from our discussions that very few platforms would be able to support unified today, because of these ordering concerns. However, there's a couple things to note: 1) there have been platforms where UNIFIED would work (and, arguably, x86_64 still provides the required ordering to make UNIFIED work 2) Just because we hang NICs off the dumbest interface possible (PCIe) today does not mean we will continue to do so. A NIC sitting off the right parts of the processor could, with the right design, provide UNIFIED even on a processor with aggressive reordering by being properly integrated into the reorder engine. So, I guess I really don't see a problem with what we have today. As soon as you require a target side WIN_SYNC, there's no point in using Unified; the user and the implementation can live quite happily with SEPARATE. And, I believe, make SEPARATE go fast on platforms that only need a memory barrier to ensure target side ordering, rather than a complicated cache protocol. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 04 Aug '13

04 Aug '13

Yes, that's one way of fixing this. Another way is to add a third memory model that is in between SEPARATE and UNIFIED, that requires WIN_SYNC. Sent from my iPad On Aug 4, 2013, at 9:03 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Pavan Balaji <balaji(a)mcs.anl.gov> writes: >> I guess the question here is -- when can the target process read the >> data written by the origin process? In the separate model, the answer >> is simple -- we do the necessary synchronization and then do a WIN_SYNC >> to synchronize the two windows. In the unified model, it's unclear >> since the user does not need to a WIN_SYNC. > > This is lack of clarity in the standard, but I think not lack of clarity > in practice (to a user familiar with memory fences), provided that > certain functions (like MPI_Win_sync) contain suitable memory fences. > > I don't think it's worth talking about x86 specifically. Rather, I > would define semantics in terms of a weak memory model (C11 or > Alpha/Linux kernel). On architectures with stronger memory models, some > fences become no-ops, but that doesn't have to be discussed in the > standard.

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jed Brown 04 Aug '13

04 Aug '13

Pavan Balaji <balaji(a)mcs.anl.gov> writes: > I guess the question here is -- when can the target process read the > data written by the origin process? In the separate model, the answer > is simple -- we do the necessary synchronization and then do a WIN_SYNC > to synchronize the two windows. In the unified model, it's unclear > since the user does not need to a WIN_SYNC. This is lack of clarity in the standard, but I think not lack of clarity in practice (to a user familiar with memory fences), provided that certain functions (like MPI_Win_sync) contain suitable memory fences. I don't think it's worth talking about x86 specifically. Rather, I would define semantics in terms of a weak memory model (C11 or Alpha/Linux kernel). On architectures with stronger memory models, some fences become no-ops, but that doesn't have to be discussed in the standard.

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 03 Aug '13

03 Aug '13

We say that the public and private windows are the same. We also say that after a flush the data is available in the target public window. I guess the question here is -- when can the target process read the data written by the origin process? In the separate model, the answer is simple -- we do the necessary synchronization and then do a WIN_SYNC to synchronize the two windows. In the unified model, it's unclear since the user does not need to a WIN_SYNC. -- Pavan On 08/03/2013 07:02 PM, Underwood, Keith D wrote: > Do we make a statement about the state of the memory? Or, do we make a statement about the ordering of loads at the target? It seems that we make the former statement and not the latter. We similarly cannot guarantee that the compiler won't optimize away a loop spinning on an RMA variable if the user doesn't do something. > > Keith > >> -----Original Message----- >> From: Pavan Balaji [mailto:[email protected]] >> >> While that statement is true in general, some of us thought that those "some >> places" included all cache-coherent architectures, and not just x86-like >> architectures. Clearly that was not what *all* of the WG thought. >> >> Basically, the situation right now is that other cache coherent architectures >> that require a memory barrier on the target side cannot support UNIFIED at >> all (at least, not without considerable overhead). >> This means that other use cases of UNIFIED, such as concurrently accessing >> nonoverlapping portions of memory through load/store and PUT/GET, are >> not available to users of such architectures either (even though the >> hardware is perfectly capable of providing this ability). >> >> In short -- >> >> SEPARATE is too general. It doesn't take advantage of some hardware >> features provided by cache-coherent hardware. >> >> UNIFIED is too specific. It is too hard for non-x86 architectures to provide. >> >> -- Pavan >> >> On 08/02/2013 09:56 PM, Underwood, Keith D wrote: >>> The point of UNIFIED was that we add a more useful model that worked in >> *some* places. >>> >>>> -----Original Message----- >>>> From: Pavan Balaji [mailto:[email protected]] >>>> Sent: Friday, August 02, 2013 7:30 PM >>>> To: Underwood, Keith D >>>> Cc: Jed Brown; mpi3-rma(a)lists.mpi-forum.org >>>> Subject: Re: [Mpi3-rma] MPI-3 UNIFIED model clarification >>>> >>>> >>>> On 08/02/2013 08:27 PM, Underwood, Keith D wrote: >>>>> That was specifically *not* the point of unified. The point of >>>>> unified was that "if the platform can support it, the user can get >>>>> something better". Remember all those conversations about >>>>> synchronization using hands waving across the machine room? Unless >>>>> you guys want to go get rid of UNIFIED? There really isn't a point >>>>> to unified if you have to call MPI_Win_sync, is there? >>>> >>>> Yes, I remember that discussion. However, what I took away from that >>>> discussion was that the user *could* do something outside of MPI, not >>>> *must*, to ensure correct behavior. >>>> >>>> If that was not the intention, then there needs to be a memory model >>>> where that's the case. >>>> >>>> -- Pavan >>>> >>>> -- >>>> Pavan Balaji >>>> http://www.mcs.anl.gov/~balaji >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Underwood, Keith D 03 Aug '13

03 Aug '13

Do we make a statement about the state of the memory? Or, do we make a statement about the ordering of loads at the target? It seems that we make the former statement and not the latter. We similarly cannot guarantee that the compiler won't optimize away a loop spinning on an RMA variable if the user doesn't do something. Keith > -----Original Message----- > From: Pavan Balaji [mailto:[email protected]] > > While that statement is true in general, some of us thought that those "some > places" included all cache-coherent architectures, and not just x86-like > architectures. Clearly that was not what *all* of the WG thought. > > Basically, the situation right now is that other cache coherent architectures > that require a memory barrier on the target side cannot support UNIFIED at > all (at least, not without considerable overhead). > This means that other use cases of UNIFIED, such as concurrently accessing > nonoverlapping portions of memory through load/store and PUT/GET, are > not available to users of such architectures either (even though the > hardware is perfectly capable of providing this ability). > > In short -- > > SEPARATE is too general. It doesn't take advantage of some hardware > features provided by cache-coherent hardware. > > UNIFIED is too specific. It is too hard for non-x86 architectures to provide. > > -- Pavan > > On 08/02/2013 09:56 PM, Underwood, Keith D wrote: > > The point of UNIFIED was that we add a more useful model that worked in > *some* places. > > > >> -----Original Message----- > >> From: Pavan Balaji [mailto:[email protected]] > >> Sent: Friday, August 02, 2013 7:30 PM > >> To: Underwood, Keith D > >> Cc: Jed Brown; mpi3-rma(a)lists.mpi-forum.org > >> Subject: Re: [Mpi3-rma] MPI-3 UNIFIED model clarification > >> > >> > >> On 08/02/2013 08:27 PM, Underwood, Keith D wrote: > >>> That was specifically *not* the point of unified. The point of > >>> unified was that "if the platform can support it, the user can get > >>> something better". Remember all those conversations about > >>> synchronization using hands waving across the machine room? Unless > >>> you guys want to go get rid of UNIFIED? There really isn't a point > >>> to unified if you have to call MPI_Win_sync, is there? > >> > >> Yes, I remember that discussion. However, what I took away from that > >> discussion was that the user *could* do something outside of MPI, not > >> *must*, to ensure correct behavior. > >> > >> If that was not the intention, then there needs to be a memory model > >> where that's the case. > >> > >> -- Pavan > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 03 Aug '13

03 Aug '13

While that statement is true in general, some of us thought that those "some places" included all cache-coherent architectures, and not just x86-like architectures. Clearly that was not what *all* of the WG thought. Basically, the situation right now is that other cache coherent architectures that require a memory barrier on the target side cannot support UNIFIED at all (at least, not without considerable overhead). This means that other use cases of UNIFIED, such as concurrently accessing nonoverlapping portions of memory through load/store and PUT/GET, are not available to users of such architectures either (even though the hardware is perfectly capable of providing this ability). In short -- SEPARATE is too general. It doesn't take advantage of some hardware features provided by cache-coherent hardware. UNIFIED is too specific. It is too hard for non-x86 architectures to provide. -- Pavan On 08/02/2013 09:56 PM, Underwood, Keith D wrote: > The point of UNIFIED was that we add a more useful model that worked in *some* places. > >> -----Original Message----- >> From: Pavan Balaji [mailto:[email protected]] >> Sent: Friday, August 02, 2013 7:30 PM >> To: Underwood, Keith D >> Cc: Jed Brown; mpi3-rma(a)lists.mpi-forum.org >> Subject: Re: [Mpi3-rma] MPI-3 UNIFIED model clarification >> >> >> On 08/02/2013 08:27 PM, Underwood, Keith D wrote: >>> That was specifically *not* the point of unified. The point of >>> unified was that "if the platform can support it, the user can get >>> something better". Remember all those conversations about >>> synchronization using hands waving across the machine room? Unless >>> you guys want to go get rid of UNIFIED? There really isn't a point to >>> unified if you have to call MPI_Win_sync, is there? >> >> Yes, I remember that discussion. However, what I took away from that >> discussion was that the user *could* do something outside of MPI, not >> *must*, to ensure correct behavior. >> >> If that was not the intention, then there needs to be a memory model where >> that's the case. >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0